ext4: No space left on device, but df -h shows free space

What you see

Writes fail, but df -h insists there is room:

$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        59G   41G   15G  74% /var

$ echo hello > /var/spool/postfix/maildrop/test
-bash: /var/spool/postfix/maildrop/test: No space left on device

Nothing was ever deleted to make room, and du -sh /var agrees there is space. The filesystem is not out of blocks — it is out of inodes, the fixed-size records that hold a file’s metadata. Every file, directory and symlink consumes exactly one inode, no matter how small it is.

Why the inode count is fixed

ext4 allocates its inode table at mkfs time. The default ratio is one inode per 16 KiB of space, so a 100 GB volume gets about 6.5 million inodes — plenty for a web server, immediately exhausted by a mail queue full of 700-byte files or a Docker build cache.

You cannot add inodes to an existing ext4 filesystem. tune2fs can show the count and change a lot of things, but not that one:

$ sudo tune2fs -l /dev/sda2 | grep -i inode
Inode count:              1472
Free inodes:              1460
Inodes per group:         184
Inode size:               256

Inode count is fixed for the life of the filesystem, and Inodes per group × number of groups is where it came from — set by the ratio at mkfs time.

One trap: on a mounted filesystem tune2fs -l reads the on-disk superblock, which lags behind the live counters. In the test below it still reports 1460 free inodes while df -i reports zero. Trust df -i; if you want a consistent tune2fs reading, umount first.

XFS allocates inodes dynamically, which is why mail relays and build machines usually run it.

Reproduce it in 60 seconds

A loop-mounted image, so nothing on the host is touched:

sudo -i
dd if=/dev/zero of=/root/tiny.img bs=1M count=64 status=progress
mkfs.ext4 -N 1500 -F /root/tiny.img
mkdir -p /mnt/tiny
mount -o loop /root/tiny.img /mnt/tiny

for i in $(seq 1 2000); do
  head -c 1024 /dev/zero > /mnt/tiny/f$i 2>/dev/null || { echo "FAILED at file $i"; break; }
done

df -h /mnt/tiny
df -i /mnt/tiny
echo hello > /mnt/tiny/newfile

On Debian 13 with e2fsprogs 1.47.2 this is what you get. mkfs rounds -N 1500 down to a valid per-group number and reports 1472 inodes:

Creating filesystem with 65536 1k blocks and 1472 inodes

After the loop dies at file 1461 — twelve inodes were already spoken for by the filesystem itself — the two df calls disagree, and that disagreement is the diagnosis:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       59M  1.6M   53M   3% /mnt/tiny

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop0       1472  1472     0  100% /mnt/tiny

53 MB available, 3% used, and not one more file fits. The error arrives from bash itself, before any data is written:

-bash: /mnt/tiny/newfile: No space left on device

In a real incident the same message comes from Postfix, PHP, or a container runtime, and it is easy to misread as a full disk. df -i settles it in one line.

The same test on XFS

Rocky Linux 9 uses XFS for /, so the loop test behaves completely differently there. A 512 MiB image — the smallest mkfs.xfs will accept — with the identical write loop:

mkfs.xfs -f -d size=512m /root/xfs.img
mount -o loop /root/xfs.img /mnt/xfs

for i in $(seq 1 5000); do
  head -c 1024 /dev/zero > /mnt/xfs/f$i 2>/dev/null || { echo "FAILED at $i"; break; }
done

All 5000 files are created and nothing fails (column headers translated from the Chinese locale the test machine used):

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      448M   57M  392M  13% /mnt/xfs

Filesystem     Inodes IUsed  IFree IUse% Mounted on
/dev/loop0     262144  5003 257141   2% /mnt/xfs

The mkfs.xfs output explains those numbers: isize=512 imaxpct=25. Inodes are allocated in chunks on demand, capped at 25% of the data area, so for a 512 MiB filesystem the ceiling works out to 262,144 inodes — about 2 KB of space per inode. On XFS you exhaust blocks long before inodes, which is why mail relays, CI builders and container hosts are usually formatted XFS.

Find what ate the inodes

Go by inode count, not by size. GNU du counts inodes with --inodes, which is far cheaper than find | wc -l:

sudo du -x --inodes -d1 /var 2>/dev/null | sort -n | tail -10

On the test filesystem above the same command points straight at the culprit — one directory, 1462 inodes, no size worth mentioning:

# du -x --inodes -d1 /mnt/tiny | sort -n | tail -3
1	/mnt/tiny/lost+found
1462	/mnt/tiny

Then drill into the winner until the count drops to something you recognise:

sudo du -x --inodes -d1 /var/spool 2>/dev/null | sort -n | tail -10
sudo du -x --inodes -d1 /var/spool/postfix/maildrop | sort -n | tail -5

The usual suspects, in the order they actually show up:

Counting files in a directory with millions of entries is itself slow — that is a symptom worth noting, because ls hanging is the second clue after df -i.

Fix it now

Free inodes by deleting files, smallest risk first. Never rm -rf a spool directory: delete its contents so the daemon keeps working.

# mail queue: inspect before deleting
sudo postqueue -p | tail -5
sudo find /var/spool/postfix/maildrop -type f -mtime +2 -delete

# php sessions
sudo find /var/lib/php/sessions -type f -mtime +1 -delete

# caches that are safe to drop
sudo npm cache clean --force
sudo apt-get clean
sudo docker system prune -af

Check progress with df -i after each step — the percentage drops immediately, and writes start working again well before the disk looks emptier.

Fix it structurally

If the workload legitimately creates millions of small files (mail relay, CI builder, container host with heavy layer churn), deleting is a weekly chore. Two real options:

1. Recreate the filesystem with a denser inode ratio. mkfs.ext4 -i 8192 gives one inode per 8 KiB instead of 16 KiB — double the inodes. mkfs.ext4 -T small does the same thing with a preset. This requires a fresh filesystem, so it is a planned migration, not a fix.
2. Put that directory on XFS. Inodes are allocated on demand, so this class of failure disappears. Good for /var/spool, /var/lib/docker, and anything that builds source trees.

For everything else, the ratio is fine and the problem is a leak: something creates files faster than it removes them.

Prevent the next one

df -h monitoring misses this failure entirely, which is why it always arrives as a surprise. Monitor inodes too:

df -i | awk 'NR>1 && $5+0 >= 80 {print $6, $5}'

Put that in cron and alert on it, or use the same check in Zabbix/Telegraf (vfs.fs.inode). 80% on a filesystem with millions of inodes means something is leaking, and you still have room to investigate calmly.

Verify

df -i /var

IUse% back under control and a test write succeeds:

echo ok | sudo tee /var/spool/postfix/maildrop/.write-test && sudo rm /var/spool/postfix/maildrop/.write-test

Related

• Docker: permission denied on /var/run/docker.sock
• systemd: Failed to Start Service (203/EXEC)
• nginx 502: upstream prematurely closed connection