Troubleshooting guide · Evicted pods

Evicted pods: why the kubelet removed them, and how to stop it

An Evicted pod was stopped by the kubelet because its node ran short of memory, disk or process IDs. The kubelet evicts to protect the node, and it chooses first the pods using more than they requested. The pod's message names the resource and the numbers, and requests set to real usage are what keep a workload off the list.

Updated

Find them, and read why

Evicted pods are left behind as Failed pods:

kubectl get pods -A --field-selector=status.phase=Failed

The message names the resource: The node was low on resource: memory, followed by what the container used against its request:

kubectl describe pod <pod> -n <namespace>

The node's conditions — MemoryPressure, DiskPressure, PIDPressure — at the time and now:

kubectl describe node <node>

How the kubelet chooses

  • Pods using more of the scarce resource than they requested go first. A pod with no request at all — BestEffort — is always over it.
  • Among those, lower priority goes before higher, and then whoever is furthest over their request.
  • A pod using no more than it requested is evicted only when nothing else frees enough.
  • A container writing past its ephemeral-storage limit is evicted on its own, whatever the node's state, with a message saying so.

Stop it happening

  • Set requests close to real use, for memory especially. The scheduler places pods by request, and a node full of pods requesting less than they use is a node heading for pressure.
  • Give workloads that must survive a memory request equal to their limit, and a PriorityClass above the rest.
  • For disk pressure, find what is filling the node: container logs, emptyDir volumes, images. Set ephemeral-storage requests and limits on pods that write locally.

Clean up

Evicted pods stay as Failed objects until the control plane's garbage collector removes them, and on most clusters that only happens after thousands have piled up. They use no resources, but they clutter every list:

Delete a namespace's Failed pods — including any failed Job pods you meant to keep for their logs:

kubectl delete pods -n <namespace> --field-selector=status.phase=Failed

In PodSteer

What PodSteer does about it.

  • An evicted pod is a critical finding on the overview, quoting the kubelet's own message about what the node ran short of.
  • Nodes under memory, disk or PID pressure are findings too, one per condition.
  • A pod with no requests is flagged as first in the queue to be evicted, before it happens.
  • The capacity panel measures CPU and memory against requests, which is where eviction risk starts.
PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means. PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means.
Findings, named by the workload. Each finding names the workload it belongs to and says what the state means, before a single pod has been opened.
Download PodSteer Cluster assessment

Read next

OOMKilled

The container used more memory than its limit and the kernel killed it. Confirm it, measure the real peak, then size the limit or fix the leak.

Node NotReady

The kubelet stopped reporting, or reported it cannot run pods. Its pods keep showing Running until they are evicted.

Pod stuck in Pending

No node can take the pod. The scheduler's FailedScheduling event names every constraint that ruled a node out.