Kubernetes: container terminated with OOMKilled (exit code 137)

Hi, it's BlueByte. A pod runs fine for twenty minutes, restarts, runs again, restarts again, and kubectl describe pod shows Reason: OOMKilled with Exit Code: 137. The application log ends mid-sentence, because nothing in the app chose to exit — the kernel killed it. We'll walk through what the status means, the three different things that end in the same 137, the commands that tell them apart, the fix for each, and how to confirm the pod stays up.

What OOMKilled and 137 are telling you

Exit code 137 is 128 + 9: the process died from signal 9, SIGKILL. The OOMKilled reason narrows that to the kernel's out-of-memory killer. It shows up in three shapes depending on when you look:

NAME            READY   STATUS      RESTARTS   AGE
memory-demo-2   0/1     OOMKilled   1          24s

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Ready:          False
    Restart Count:  5

The third shape is CrashLoopBackOff in kubectl get pods with the block above under Last State. All three are the same event. The Kubernetes docs put the mechanism in one sentence: "memory limits are enforced by the kernel with out of memory (OOM) kills", and add that the kill happens when the kernel detects pressure, so a container can sit above its limit briefly before it dies. Nothing is broken in Kubernetes here; it did what the limit asked.

Three different killers that end in the same status

Three causes, three fixes.

The container's own limit is too small. The limit is the memory cgroup's ceiling. A service whose real working set is 900 MiB under limits.memory: 512Mi is killed every time traffic pushes it over. Watch for the typo the docs call out: memory: 400m means 400 millibytes — 0.4 bytes — when the author meant 400Mi.

The runtime sized its heap without looking at the limit. A JVM started with -Xmx1g inside a 768Mi container dies as soon as the heap grows, because heap plus metaspace plus thread stacks exceed the cgroup. Node has the same shape with --max-old-space-size. A leak belongs here too: usage climbs for hours, then hits the ceiling.

The node ran out, not the container. With no limit, the container is only killed when the whole node is short. The kubelet sets oom_score_adj per QoS class — Guaranteed −997, BestEffort 1000, Burstable in between based on the request — so a pod with no requests is the first thing the kernel's oom_killer picks.

Find out which one you have before you change anything

Read the last termination and the resources in one go:

kubectl get pod api-7d9f -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.restartCount}{"\n"}{end}'
kubectl get pod api-7d9f -o jsonpath='{.spec.containers[*].resources}'

api	OOMKilled	137	5
{"limits":{"memory":"512Mi"},"requests":{"memory":"512Mi"}}

Then check live usage. kubectl top needs metrics-server; Metrics API not available means that add-on is missing:

kubectl top pod api-7d9f --containers

POD        NAME   CPU(cores)   MEMORY(bytes)
api-7d9f   api    120m         498Mi

Usage creeping toward the limit before each restart means cause one or two. No limit on the container, plus a Warning event of reason SystemOOM on the node — the kubelet records "System OOM encountered, victim process: java, pid: 1234" — means the node was the one out of memory:

kubectl get events -A --field-selector reason=SystemOOM

Fix 1: set the limit from the measured working set, with headroom

Watch kubectl top through a normal peak, set the limit about 25–30% above the highest value, and set the request to what the pod needs at rest. Requests decide scheduling; limits decide the kill line:

resources:
  requests:
    memory: "768Mi"
  limits:
    memory: "1Gi"

Two details from the docs: set only a limit and Kubernetes copies it into the request, so the scheduler reserves the full limit; and suffixes are case-sensitive — Mi mebibytes, M megabytes, m millibytes.

Fix 2: make the heap fit inside the limit

Modern JVMs are container-aware: the default maximum heap is 25% of the container's memory (-XX:MaxRAMPercentage defaults to 25), read from the cgroup ceiling, not the node's RAM. Raise the percentage rather than hard-coding -Xmx, and the heap follows the limit:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75.0"

For Node, --max-old-space-size is in MiB and bounds only the old generation, so leave room below the limit:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=768"
resources:
  limits:
    memory: "1Gi"

If usage still climbs without bound, you have a leak, and a larger limit only delays the kill — take a heap dump near the high-water mark and fix the app.

Fix 3: when the node was the one out of memory

Give the pod requests so it stops being BestEffort. Requests equal to limits make it Guaranteed, with oom_score_adj −997, at the back of the kernel's list. It still dies if it alone exceeds its limit — that is cause one — but it stops paying for a neighbour's leak.

kubectl get pod api-7d9f -o jsonpath='{.status.qosClass}'

Guaranteed is the value you want afterwards. If many pods on the node are BestEffort, a LimitRange with default requests fixes the whole namespace at once.

A real case: a Java API killed every forty minutes after a limit cut

A team cut limits.memory on an API from 2Gi to 1Gi to pack more pods per node. From then on it restarted about every forty minutes with OOMKilled. kubectl top pod --containers read 990Mi just before each restart, and the deployment still carried JAVA_TOOL_OPTIONS=-Xmx1536m from the 2Gi days — a heap allowed to grow past the new ceiling. They replaced the flag with -XX:MaxRAMPercentage=70.0, set the request to 768Mi, and kept the 1Gi limit. The next rollout held at about 740Mi under load and Restart Count stayed at 0 all week.

Check it end to end and keep it from coming back

Verify with the same commands you diagnosed with, a while after the fix:

kubectl get pod -l app=api -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'
kubectl top pod -l app=api --containers

A restart count that stops climbing and usage that plateaus below the limit is the proof. To prevent a repeat: put a LimitRange in every namespace so no pod runs without requests, load-test before cutting a limit, alert on the kubelet's container_oom_events_total metric so you see the first kill rather than the fiftieth, and treat usage that grows for days as a leak to fix, not a limit to raise.

How this differs from Evicted and from a plain 137

Evicted is the kubelet, not the kernel: when the node's memory.available signal drops under an eviction threshold, the kubelet ends pods itself, the pod status reads Evicted, and the message says "The node was low on resource: memory". Which pods go first depends on requests and usage, not a cgroup ceiling, so the fix is cause three's — set requests — plus more room on the node. A 137 without OOMKilled (Reason: Error) is a SIGKILL from somewhere else, most often the kubelet force-stopping a container that ignored SIGTERM past its grace period after a failed liveness probe. Next time you see 137, read the reason first: OOMKilled sends you to memory, Error sends you to probes and shutdown handling.