Troubleshooting guide · Node NotReady

Node NotReady: when a Kubernetes node stops reporting

A node is NotReady when its kubelet has stopped reporting to the control plane, or reports that it cannot run pods — the runtime is down, the network plugin is not ready, or the node is out of resources. Its pods go on showing Running for a while, which is why the rest of the cluster looks healthier than it is.

Updated

What happens to the pods

When a node stops reporting, the control plane marks it NotReady or Unknown and taints it, so nothing new is scheduled there. Pods already on it keep their last reported status — usually Running — and are evicted only when their toleration for that taint runs out, five minutes by default. Controllers then replace them elsewhere; a StatefulSet pod waits until the node is confirmed gone.

Read what the node last said

Which nodes are not Ready, and for how long:

kubectl get nodes

The Conditions table: the Ready condition's reason and message, and the pressure conditions beside it:

kubectl describe node <node>

A Ready condition of Unknown with the message Kubelet stopped posting node status means the control plane has heard nothing — the kubelet is down or unreachable. False with a message means the kubelet is talking and says why it cannot run pods.

The common causes

  • The kubelet stopped or keeps crashing. On the node, systemctl status kubelet and journalctl -u kubelet say why — often an expired client certificate or a configuration change.
  • The container runtime is down. The kubelet reports it cannot reach containerd or CRI-O; restart the runtime and read its logs.
  • The network plugin is not ready: the message names the CNI — cni plugin not initialized, or no configuration found. Check the CNI's own pods, usually a DaemonSet in kube-system.
  • The node is out of memory, disk or PIDs, and the kubelet itself is starved. The pressure conditions say which.
  • The node is unreachable: a network partition, or a machine that was stopped or deleted underneath the cluster.

While you fix it

Keep new pods off the node while you work on it:

kubectl cordon <node>

Move what is left once the node is back, respecting disruption budgets:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

Remove a node that is gone for good, so the pods waiting on it can be replaced:

kubectl delete node <node>

In PodSteer

What PodSteer does about it.

  • Nodes that are not ready are a critical finding, with the warning that pods on them keep reporting Running until the eviction timeout — so every other number is optimistic until it is fixed.
  • Memory, disk and PID pressure are findings of their own, one per condition.
  • Cordon and drain are one action each, and a drain shows its per-pod plan — what is evicted, what is skipped and what a budget would refuse — before it runs.
PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means. PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means.
Findings, named by the workload. Each finding names the workload it belongs to and says what the state means, before a single pod has been opened.
Download PodSteer Cluster assessment Day-2 operations

Read next

Pod stuck in Terminating

The pod was deleted but has not gone. A finalizer, a process ignoring SIGTERM, or a node nobody can reach is holding it.

Evicted pods

A node ran short of memory, disk or process IDs and the kubelet reclaimed it by evicting pods — first those using more than they requested.

Pod stuck in Pending

No node can take the pod. The scheduler's FailedScheduling event names every constraint that ruled a node out.