Kubernetes pod stuck in CrashLoopBackOff
Hi, it's BlueByte. CrashLoopBackOff looks alarming, but it is just Kubernetes telling you a container keeps exiting right after it starts. It is a symptom, not a cause. Let's read the two things that actually name the cause — the previous container's logs and its exit code — and fix the real problem instead of guessing.
What CrashLoopBackOff is (and isn't)
A pod never reaches Running:
NAME READY STATUS RESTARTS AGE
api-7d9f 0/1 CrashLoopBackOff 5 3mThe container starts, exits, and the kubelet restarts it — backing off longer each time (10s, 20s, 40s, up to five minutes). The status is the loop, not the reason. The reason is whatever makes the container exit, and it is in the logs and the exit code.
The handful of causes, each with a fingerprint
The main process is terminating shortly after start. The common reasons, each recognizable:
- Application error on startup — a missing env var, an unreachable dependency (a database that is not up yet), or bad config. Exit code usually
1or2, with a stack trace in the logs. - Out of memory — the container exceeded its memory limit and was killed. Exit code
137(SIGKILL), reasonOOMKilled. - A liveness probe that fails because the app is slow to start.
- A wrong command or entrypoint — the process exits immediately or the binary is not found, often with empty logs.
- A missing volume or secret, so the app cannot read its config and bails out.
Read the previous container's logs and exit code
This is the whole job. Read the instance that already crashed, not the one starting now:
kubectl logs api-7d9f --previousThen the exit code and reason:
kubectl describe pod api-7d9fLook at Last State: Terminated. 137 with OOMKilled is memory; 1/2 is an application error the log explains; a Liveness probe failed event points at the probe; and the Events list calls out a missing ConfigMap, Secret, or mount.
Fix the specific cause
- Application error: set the missing env var, correct the config, or make the dependency reachable, then let the pod restart.
- OOMKilled (137): raise the memory limit or reduce usage:
resources:
limits:
memory: "512Mi"- Liveness probe too strict: give the app time to start:
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
initialDelaySeconds: 30A real case: a missing env var
kubectl get pods shows api-7d9f at CrashLoopBackOff with six restarts. kubectl logs api-7d9f --previous prints Error: DATABASE_URL is not set and nothing else, and kubectl describe pod confirms Exit Code: 1. The cause is clear — not memory, not the probe. You add DATABASE_URL to the deployment's env (or the referenced Secret), apply it, and the next restart reaches Running with the count frozen. Reading the previous log first saved you from raising memory limits that were never the problem.
Confirm it settles
Watch the pod:
kubectl get pod api-7d9f -wIt should move to Running at READY 1/1 and stop incrementing RESTARTS. If restarts keep climbing, the log names a cause you have not addressed yet.
How this differs from ImagePullBackOff and Pending
ImagePullBackOff looks similar in kubectl get pods but the image never started — the name or registry credentials are wrong. Pending means the pod has not been scheduled (no node has room). CrashLoopBackOff means the container did run and then exited, so it is the only one where the logs and exit code are your map. Set memory limits deliberately, keep probes generous, and wait for dependencies with an init container so the pod holds instead of looping.
Related questions
What does exit code 137 mean?
The container was killed with SIGKILL, almost always an out-of-memory kill. Raise the memory limit or reduce usage.
kubectl logs --previous says there is no previous container.
The container is failing before it writes anything — check the command/args and image entrypoint, which run before your app's logging starts.
The restart count keeps climbing. Is that harmful?
The back-off caps at five minutes between restarts, so it will not hammer the node, but the pod serves no traffic until the underlying cause is fixed.
How is this different from ImagePullBackOff?
ImagePullBackOff means the image could not be pulled (bad name or credentials). CrashLoopBackOff means the image ran and the process exited.
The app needs a database that starts later. How do I stop the crash loop?
Use an init container or a readiness probe to wait for the dependency instead of letting the app crash on it, so the pod holds rather than loops until the database is up.
References
Haneul Seo
Infrastructure engineer · 10+ years running Linux fleets
More in this category
Git: fatal: detected dubious ownership in repository
Since the CVE-2022-24765 fix in Git 2.35.2, Git refuses to read a repository whose working tree or .git directory is owned by a different user than the one running the command. It shows up in containers, CI jobs, sudo sessions and shared drives. Fix the ownership if the repo should be yours, or add the exact path to safe.directory in your global config — never in the repo's own config, which Git ignores for this.
Kubernetes: Internal error occurred: failed calling webhook
An admission webhook sits in front of your write, the API server could not get an answer out of it, and failurePolicy: Fail turned that silence into a rejection. The tail of the message is the whole diagnosis: context deadline exceeded means the call went nowhere, no endpoints available means nothing is running, and an x509 line means the API server does not trust the webhook's certificate. Each has a different fix, and none of them is your manifest.
MySQL: ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
A statement waited the full innodb_lock_wait_timeout for a row lock another transaction is still holding, and gave up. sys.innodb_lock_waits names the blocking session and hands you the KILL statement, and a blocking_query of NULL means the blocker is idle on an open transaction. The detail most retry loops get wrong: by default only the timed-out statement is rolled back, so your transaction is still open and still holds every lock it took earlier.
Redis: MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk
Reads keep working and every write is rejected, because the last background save failed and stop-writes-on-bgsave-error defaults to yes. The log names the real cause — no space, a dir the redis user can't write, a read-only mount at rename time, or fork failing with Cannot allocate memory. Fix the cause, run one BGSAVE, and writes come back on their own with no restart: rdb_last_bgsave_status flips from err to ok. Setting stop-writes-on-bgsave-error no restores writes instantly but leaves the snapshot broken, so treat it as a deliberate trade, not the fix.
Node.js: FATAL ERROR: Reached heap limit — JavaScript heap out of memory (exit 134)
The V8 heap has its own ceiling, derived from system memory and the Node release, and it is often far below the RAM you have; when a build or server reaches it, V8 aborts with FATAL ERROR: Reached heap limit and exit code 134. Read the real limit with v8.getHeapStatistics().heap_size_limit, then raise it with --max-old-space-size (in MiB) or NODE_OPTIONS for a large workload, size it below the cgroup limit inside containers, and use --heapsnapshot-near-heap-limit to catch a leak in a long-running process. Exit 137 with no FATAL ERROR line is a container kill, not this.
Docker: "exec format error" when the container starts — wrong-platform image, no emulator, or a script with no shebang
The container exits on its first instruction with exec format error — the kernel's ENOEXEC, meaning the file exists but cannot be executed here. In practice that is an image built on one CPU architecture (an Apple-silicon Mac produces linux/arm64) and run on another (an x86_64 server) with no QEMU handler registered in binfmt_misc, or an entrypoint script whose first line is not a shebang. uname -m, docker image inspect and ls /proc/sys/fs/binfmt_misc tell the causes apart; the fix is an explicit docker buildx build --platform (or a manifest list for both), QEMU registration or --platform when you mean to emulate, and a #!/bin/sh line for the script.