Troubleshooting guide · OOMKilled

OOMKilled: when Kubernetes kills a container for memory

OOMKilled means a container used more memory than its limit allowed, and the kernel killed it with SIGKILL — which is why it exits with code 137. The fix is either a limit that matches the container's real peak, or an application that uses less; raising the limit blindly only moves the problem.

Updated 2026-09-25

What happened

A container's memory limit is enforced by the kernel. When the processes inside it try to use more than the limit, the kernel's out-of-memory killer stops one of them with SIGKILL, and the kubelet records the container's termination reason as OOMKilled with exit code 137 (128 + 9). If the pod restarts it and it happens again, the pod goes into CrashLoopBackOff.

It is not the same as an eviction. When a whole node runs short of memory, the kubelet evicts pods to protect the node, and those pods end with the reason Evicted. OOMKilled is about one container and its own limit.

Confirm it

Last State should read Terminated, Reason: OOMKilled, Exit Code: 137:

kubectl describe pod <pod> -n <namespace>

The same fact, for every container in the pod, without the rest of describe:

kubectl get pod <pod> -n <namespace> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\n"}{end}'

The requests and limits the container was given:

kubectl get pod <pod> -n <namespace> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}'

Exit code 137 without OOMKilled

An exit code of 137 without the OOMKilled reason is a different problem: the container was killed, but not for memory — typically a liveness probe, or a shutdown that outlasted its grace period. Raising the memory limit will not help.

Measure what it really uses

A limit should sit above the container's real peak, and the only way to know the peak is to measure it under realistic load. With metrics-server installed:

kubectl top shows a moment, not a peak. Watch it through the busy part of the day, or read the container_memory_working_set_bytes series from your monitoring system, which is the figure the limit is enforced against.
Memory that climbs steadily and never falls is a leak, and no limit fixes it — the container is killed later instead of sooner.

Current memory per container:

kubectl top pod <pod> -n <namespace> --containers

Fix it

Raise the limit to above the measured peak, with headroom for the busiest hour you have seen.
Set the request close to real use as well. The scheduler places pods by request, so a request far below use packs pods onto a node that cannot hold them all.
Tell the runtime about the limit. A JVM sizes its heap from -XX:MaxRAMPercentage, Node.js from --max-old-space-size, and a runtime that thinks it has the whole machine will grow past a container's limit.
Fix what is holding memory: unbounded caches, queues that grow when a consumer falls behind, whole files or responses read into memory.
For a workload that must not be killed for memory, give it a memory request equal to its limit, which also makes it one of the last candidates for eviction when the node is short.

In PodSteer

What PodSteer does about it.

A container killed for memory is a finding on the overview, with the termination explained rather than printed as an exit code.
137 is called OOMKilled only when the kubelet said so. A SIGKILL for any other reason is reported as a kill, so nobody is sent to raise a limit that was never the problem.
Sizing findings flag workloads that run over their memory request, and the detail panel draws each container's request and limit across its usage.

A list of deployments in the storefront namespace with checkout open in a detail panel: three replicas desired, current, ready and available, then the pod template with the nginx image, CPU and memory requests and limits, probes, mounts and six environment variables with their values, including the pod's namespace and version, a currency, and a database password shown masked. — **The detail panel, in the manifest's order.** Environment values are shown as the container sees them, and a password stays masked.

Download PodSteer Cluster assessment