Docker: no space left on device
Hi, it's BlueByte. "No space left on device" from Docker usually means old images and build cache have quietly piled up — but not always, so let's confirm what is actually full before we start deleting. "No space" has three different meanings, and only one of them is fixed by pruning.
What the error means
A build or pull fails:
write /var/lib/docker/...: no space left on deviceDocker's storage area is full. Images, stopped containers, build cache, and unused volumes accumulate over time and are not removed automatically, so on a busy build host the space runs out. Before you prune, it is worth confirming what is actually full.
The three things that can be "full"
Everything Docker creates takes space under /var/lib/docker: every image layer, every exited container, the build cache, and orphaned volumes. Three distinct things can actually be "full", and they need different fixes:
- Reclaimable Docker data — old images, stopped containers, build cache. The common case.
- The host filesystem that
/var/lib/dockersits on is out of bytes, independent of Docker. - Inodes are exhausted even though bytes remain — many tiny files, common with layered images.
Confirm which one before deleting anything
Start with Docker's own accounting:
docker system dfIt breaks down images, containers, local volumes, and build cache with a reclaimable column. Then check the host, both bytes and inodes:
df -h /var/lib/docker
df -i /var/lib/dockerIf df -h shows space free but df -i shows 100% used, you are out of inodes, not bytes. If docker system df shows little reclaimable but df -h is full, something outside Docker is eating the disk.
Prune what you don't need
- Remove stopped containers, unused networks, dangling images, and build cache:
docker system prune- To reclaim more, also remove images not used by any container and unused volumes — this deletes data, so read the prompt:
docker system prune -a --volumes- If the host disk itself is full (not just Docker), grow the volume or clear space elsewhere — pruning Docker will not help.
A real case: a CI runner full of build cache
A CI runner's builds start failing with "no space left on device". docker system df shows 40 GB of reclaimable build cache and dangling images from weeks of jobs; df -h confirms /var/lib/docker is 98% full while df -i is fine, so it is bytes, not inodes. You run docker system prune -af (safe here — the runner keeps no persistent state), which reclaims the 40 GB, and the next build succeeds. You then add a nightly docker system prune -af so the disk never fills between jobs again.
Confirm the space is back
Re-run docker system df and df -h /var/lib/docker and confirm the space is back, then retry the build or pull that failed. It should complete without the error.
Keep it from recurring
On a CI runner, schedule a periodic docker system prune -af so the disk does not fill between jobs. Keep an eye on df -h /var/lib/docker with an alert, and be deliberate about which volumes you keep so a --volumes prune never surprises you.
How this differs from a port or memory error
no space left on device is about storage. It is not the same as port is already allocated (a networking conflict) or an OOM kill (memory). And a container that itself reports "no space left" may be hitting its own filesystem limit, not the host's — check inside the container as well. When Docker says no space, run docker system df and df -i before you prune.
Related questions
Does docker system prune delete my running containers or their data?
No. Plain prune only removes stopped containers, dangling images, and unused build cache. Running containers and their volumes are left alone unless you add --volumes and the volume is unattached.
df shows the disk is not full, but Docker says no space. Why?
You are likely out of inodes rather than bytes, or Docker's data lives on a separate, smaller filesystem. Check df -i and df -h for /var/lib/docker.
How much will --volumes remove?
Every volume not currently attached to a container, including databases you stopped but meant to keep. List them with docker volume ls first if unsure.
Is there a safe automated cleanup?
docker system prune -af on a schedule is safe for CI runners that keep no persistent state. On a host with data volumes, omit --volumes.
Can I limit how much space the build cache uses?
Yes — configure a builder with a cache size limit (docker buildx) or run docker builder prune with a keep-storage cap, so the cache is bounded instead of growing until the disk is full.
References
Haneul Seo
Infrastructure engineer · 10+ years running Linux fleets
More in this category
Git: fatal: detected dubious ownership in repository
Since the CVE-2022-24765 fix in Git 2.35.2, Git refuses to read a repository whose working tree or .git directory is owned by a different user than the one running the command. It shows up in containers, CI jobs, sudo sessions and shared drives. Fix the ownership if the repo should be yours, or add the exact path to safe.directory in your global config — never in the repo's own config, which Git ignores for this.
Kubernetes: Internal error occurred: failed calling webhook
An admission webhook sits in front of your write, the API server could not get an answer out of it, and failurePolicy: Fail turned that silence into a rejection. The tail of the message is the whole diagnosis: context deadline exceeded means the call went nowhere, no endpoints available means nothing is running, and an x509 line means the API server does not trust the webhook's certificate. Each has a different fix, and none of them is your manifest.
MySQL: ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
A statement waited the full innodb_lock_wait_timeout for a row lock another transaction is still holding, and gave up. sys.innodb_lock_waits names the blocking session and hands you the KILL statement, and a blocking_query of NULL means the blocker is idle on an open transaction. The detail most retry loops get wrong: by default only the timed-out statement is rolled back, so your transaction is still open and still holds every lock it took earlier.
Redis: MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk
Reads keep working and every write is rejected, because the last background save failed and stop-writes-on-bgsave-error defaults to yes. The log names the real cause — no space, a dir the redis user can't write, a read-only mount at rename time, or fork failing with Cannot allocate memory. Fix the cause, run one BGSAVE, and writes come back on their own with no restart: rdb_last_bgsave_status flips from err to ok. Setting stop-writes-on-bgsave-error no restores writes instantly but leaves the snapshot broken, so treat it as a deliberate trade, not the fix.
Node.js: FATAL ERROR: Reached heap limit — JavaScript heap out of memory (exit 134)
The V8 heap has its own ceiling, derived from system memory and the Node release, and it is often far below the RAM you have; when a build or server reaches it, V8 aborts with FATAL ERROR: Reached heap limit and exit code 134. Read the real limit with v8.getHeapStatistics().heap_size_limit, then raise it with --max-old-space-size (in MiB) or NODE_OPTIONS for a large workload, size it below the cgroup limit inside containers, and use --heapsnapshot-near-heap-limit to catch a leak in a long-running process. Exit 137 with no FATAL ERROR line is a container kill, not this.
Docker: "exec format error" when the container starts — wrong-platform image, no emulator, or a script with no shebang
The container exits on its first instruction with exec format error — the kernel's ENOEXEC, meaning the file exists but cannot be executed here. In practice that is an image built on one CPU architecture (an Apple-silicon Mac produces linux/arm64) and run on another (an x86_64 server) with no QEMU handler registered in binfmt_misc, or an entrypoint script whose first line is not a shebang. uname -m, docker image inspect and ls /proc/sys/fs/binfmt_misc tell the causes apart; the fix is an explicit docker buildx build --platform (or a manifest list for both), QEMU registration or --platform when you mean to emulate, and a #!/bin/sh line for the script.