Linux: "No space left on device" but df shows free space (inodes)

Hi, it's BlueByte. Your disk has gigabytes free, df -h agrees, and yet every write fails with No space left on device. Nothing is corrupt — the filesystem has simply run out of inodes, and df -h never shows that. The symptom is ENOSPC / No space left on device on a touch, a cp, a log write, or an application error, while df -h reports plenty of room. We'll walk through why free space and "no space left" can both be true at once, how to confirm it's inodes, find what's eating them, fix it, and keep it from coming back.

Why a disk with free space still says "No space left on device"

Every file on an ext4 filesystem needs two things: data blocks to hold its contents, and one inode to hold its metadata — owner, permissions, timestamps, and where the blocks live. df -h measures blocks. But the number of inodes is decided when the filesystem is created and, per the mke2fs manual, "it is not possible to change this ratio on a file system after it is created." A directory full of tiny files burns one inode each while barely touching the blocks. When the inodes run out, the kernel returns ENOSPC — the same error a full disk gives — even though df -h still shows free gigabytes. Two things also mimic this: writing to a different mount than the one you checked, and a deleted-but-still-open file — though that last one shows up as full in df -h, so it's the opposite tell.

First, ask df for inodes, not blocks

Don't trust df -h alone here — ask for inode usage with -i, which the coreutils manual defines as "list inode usage information instead of block usage":

df -h /
df -i /

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       193G  135G   58G  71% /
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/sda1      26083328 3502488 22580840   14% /

That is a healthy filesystem — 14% of inodes used, room in both columns. On an exhausted one the IUse% column reads 100% while df -h's Use% still shows space free. If IUse% is 100, you've found it. If it isn't, make sure you're even looking at the right filesystem: run df -i against the exact path that fails (df -i /var/lib/app), because the write may land on a separate mount.

Find the directory drowning in tiny files

Inodes are gone because something created a mountain of small files. Find the directory holding them by counting files per parent:

sudo find / -xdev -type f -printf '%h\n' | sort | uniq -c | sort -rn | head

4211872 /var/spool/postfix/maildrop
  38214 /var/log/journal/9f0a...
   9021 /home/app/.cache

-xdev keeps find on one filesystem so it doesn't wander into /proc or another mount. The top line — over four million files in one mail spool — is the culprit. That single directory drained the inode table.

Fix it by clearing the files that ate the inodes

Once you know the directory, delete what's safe to delete. A directory with millions of entries is too large for rm * — the shell can't expand that many arguments and errors with Argument list too long — so delete in place:

sudo find /var/spool/postfix/maildrop -type f -delete

Expected result: df -i drops right away as the inodes are freed. If the files are legitimate and you genuinely need more inodes, there is no live resize — you recreate the filesystem with a denser inode ratio using mke2fs -i <bytes-per-inode> or a fixed count with -N, then restore from backup. This applies to ext2/3/4; XFS and Btrfs allocate inodes dynamically and don't hit a fixed cap the same way, so you'd rarely see this on them.

A real case: a runaway session directory

A PHP app starts throwing failed to open stream: No space left on device, but df -h / shows 60 GB free. You run df -i / and IUse% is 100%. The file-count scan points at /var/lib/php/sessions holding six million session files that were never expired. You confirm they're stale with find /var/lib/php/sessions -type f -mtime +2 | head, then clear them with sudo find /var/lib/php/sessions -type f -mtime +2 -delete. df -i falls to 20%, and the app writes again — the blocks were never the problem.

Verify the inodes came back and writes succeed

Check the same column you started with, then prove a write works:

df -i /
touch /var/lib/app/probe && echo "write ok" && rm /var/lib/app/probe

/dev/sda1      26083328  5231044 20852284   21% /
write ok

An IUse% well under 100 and a successful touch mean the filesystem can create files again. If touch still fails, you cleared the wrong directory — re-run the file-count scan.

Keep the inode table from filling up

Expire small files before they pile up: give mail spools, session stores, and cache directories a retention policy with systemd-tmpfiles or a cron find -mtime +N -delete. Monitor df -i, not just df -h — an alert on IUse% catches this days before it stops writes. And when you build a filesystem you know will hold many small files, size the inode ratio for it at mke2fs time, since you can't change it afterward.

How this differs from a full disk and a deleted-open file

If df -h shows Use% 100%, the blocks are full — that's an ordinary disk-full, and on a container host it's usually image and log layers you clear with docker system prune, not this. If both df -h and df -i look fine yet space seems to have vanished, a process is holding a deleted file open, so its blocks aren't reclaimed until it closes — find it with sudo lsof +L1. Inode exhaustion is the only one of the three where df -h shows free space while df -i shows full.