Troubleshooting guide · CrashLoopBackOff

CrashLoopBackOff: why a pod keeps restarting, and how to fix it

CrashLoopBackOff is not an error of its own: it is the kubelet waiting before it restarts a container that keeps exiting. The reason is almost always in the logs of the attempt that just died — kubectl logs --previous — and in the exit code kubectl describe pod reports for it.

Updated

What the status means

A container in a pod whose restart policy is Always or OnFailure is restarted whenever it exits. When it keeps exiting, the kubelet waits longer before each restart — an exponential back-off that starts at ten seconds and is capped at five minutes — and while it is waiting, the pod reports CrashLoopBackOff. Once a container has run for ten minutes without trouble, the back-off resets.

So the status tells you only that the container keeps stopping. Whatever made it stop happened in the previous attempt, which is why the current container's logs are usually empty and the useful ones are one restart back.

Find out why it exits

Three commands answer it in nearly every case. Run them in this order.

The restart count, and which pods are affected:

kubectl get pods -n <namespace>

The container's Last State: its termination reason, exit code and when it finished. The Events at the bottom show probe failures and back-off messages:

kubectl describe pod <pod> -n <namespace>

The logs of the attempt that crashed, not the one waiting to start. Add -c <container> when the pod has more than one:

kubectl logs <pod> -n <namespace> --previous

Read the exit code

The exit code in Last State narrows the cause before you have read a log line. A code above 128 means the process was killed by a signal: subtract 128 to get the signal's number.

  • 1, or another small number: the application exited with an error of its own. The previous logs say which.
  • 126 or 127: the command could not be run — not executable, or not found in the image. Check command and args against what the image actually contains.
  • 137 (128 + 9, SIGKILL) with the reason OOMKilled: the container went over its memory limit. See the OOMKilled guide.
  • 137 without OOMKilled: something else sent SIGKILL — most often a liveness probe that failed, or a container that did not stop within its grace period after SIGTERM.
  • 143 (128 + 15, SIGTERM): the container was asked to stop, usually by a failing liveness probe or a deletion, and did.
  • 0: the process finished successfully. A container that is meant to keep running has run to completion — its command returns instead of serving — or it belongs in a Job rather than a Deployment.

The common causes, and their fixes

  • Missing configuration. The logs name a setting, a file or an environment variable that is not there. Add it to the ConfigMap or Secret the pod reads, or to its env, and check the names match exactly.
  • A dependency that is not ready. The application exits because a database, queue or API is unreachable at start-up. Make it retry instead of exiting, or wait for the dependency in an init container.
  • A liveness probe that fires too early. The probe fails while the application is still starting, and the kubelet kills it. Add a startupProbe, or raise the liveness probe's initialDelaySeconds and failureThreshold to cover the real start-up time.
  • Too little memory. The container is OOMKilled on start-up or under load. Raise the memory limit to above its real peak, or find what is using it.
  • The wrong command or image. A typo in command or args, an entrypoint the image does not have, or an image built for another CPU architecture.
  • Permissions. The container runs as a user that cannot write where it needs to, or a securityContext forbids something it does. The logs usually say permission denied.

Watch it recover

After a fix, a Deployment rolls out new pods on its own. For an unmanaged pod, delete it so its controller recreates it, or restart the workload:

Restart every pod of a Deployment, one batch at a time:

kubectl rollout restart deployment/<name> -n <namespace>

Watch the new pods come up:

kubectl get pods -n <namespace> -w

What PodSteer does about it.

  • A crash loop is a finding on the cluster overview the moment the cluster is opened, grouped by the workload that owns the pods — ten crashing replicas of one Deployment read as one problem, not ten.
  • The finding says where the reason is: the logs of the previous attempt, which the pod's Logs tab opens with one switch.
  • A container that exited 137 is called OOMKilled only when the kubelet also said so; otherwise PodSteer says it was killed, which is a different fix.
PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means. PodSteer's cluster overview with its findings open under a banner reading Needs attention, 4 critical and 5 warning findings. Listed: an image that cannot be pulled for two pods of storefront/reviews, two unschedulable pods of storefront/search-indexer, CrashLoopBackOff for the image-resizer deployment, an unschedulable Trivy scanner job, and the start of a warning about failing startup probes. Each finding says what the state means.
Findings, named by the workload. Each finding names the workload it belongs to and says what the state means, before a single pod has been opened.

OOMKilled

The container used more memory than its limit and the kernel killed it. Confirm it, measure the real peak, then size the limit or fix the leak.

ImagePullBackOff

The node cannot fetch the container image. The event message says whether the image is missing, the registry refused, or it could not be reached.

Pod stuck in Pending

No node can take the pod. The scheduler's FailedScheduling event names every constraint that ruled a node out.