Troubleshooting guide · Pod stuck in Terminating

Pod stuck in Terminating: what is holding the deletion

A pod shows Terminating from the moment it is deleted until the kubelet confirms its containers have stopped and every finalizer has been removed. When it stays that way for minutes, one of three things is holding it: a finalizer nothing will clear, a process that ignores the stop signal, or a node that is not there to answer.

Updated

What deletion actually does

Deleting a pod sets a deletion timestamp and a grace period — thirty seconds unless terminationGracePeriodSeconds says otherwise. The kubelet sends each container SIGTERM, waits out the grace period, then sends SIGKILL. The pod object is removed once the containers have stopped and its finalizers list is empty.

Find what is holding it

Finalizers still registered on the pod. An empty answer rules them out:

kubectl get pod <pod> -n <namespace> -o jsonpath='{.metadata.finalizers}'

Which node the pod is on:

kubectl get pod <pod> -n <namespace> -o wide

Whether that node is Ready:

kubectl get node <node>

The events, including any failure to stop a container:

kubectl describe pod <pod> -n <namespace>

What each answer means

  • A finalizer is listed: a controller has said it must clean up first — detach a volume, release a load balancer, deregister something. Find the controller that owns that finalizer and why it is not finishing; it may have been uninstalled.
  • The node is NotReady or gone: nothing is left to confirm the containers stopped, so the pod waits for a node that is not coming back. See the Node NotReady guide.
  • No finalizer, a healthy node: a process is ignoring SIGTERM and running out the grace period. A long terminationGracePeriodSeconds makes that look like a hang.

Last resorts, and what they cost

Both of these remove the pod object without the guarantee deletion normally gives. Use them only once you know why it is stuck.

Force-delete: remove the object now, without waiting for the kubelet to confirm. For a StatefulSet pod this can leave two pods with the same identity running at once:

kubectl delete pod <pod> -n <namespace> --grace-period=0 --force

Clear the finalizers by hand. Whatever the controller meant to clean up is abandoned:

kubectl patch pod <pod> -n <namespace> --type=merge -p '{"metadata":{"finalizers":null}}'

In PodSteer

What PodSteer does about it.

  • A pod still terminating after five minutes is a finding on the overview: almost always a finalizer, or a container ignoring SIGTERM.
  • The pod's panel names the finalizers holding the deletion open — the part kubectl describe leaves out.
  • Nodes that are not ready are a finding of their own, so a pod stuck on a dead node is explained by the node beside it.
PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means. PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means.
Findings, named by the workload. Each finding names the workload it belongs to and says what the state means, before a single pod has been opened.
Download PodSteer Cluster assessment Day-2 operations

Read next

Node NotReady

The kubelet stopped reporting, or reported it cannot run pods. Its pods keep showing Running until they are evicted.

Evicted pods

A node ran short of memory, disk or process IDs and the kubelet reclaimed it by evicting pods — first those using more than they requested.

CrashLoopBackOff

The container starts, exits and is restarted with a growing delay. Find the reason in the previous attempt's logs and its exit code.