Solution for No space left on device error ~ inode depletion edition

I’ll introduce the solution when No space left on device error occurs on Linux servers despite having disk space available. The cause was inode depletion.

[Prerequisites] Had disk space but inodes were depleted

Disk usage was 76% with 3.7G free space, which shouldn’t cause No space left on device errors.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2        16G   12G  3.7G  76% /
tmpfs           499M     0  499M   0% /dev/shm

I couldn’t figure out the cause and searched online to find:

While investigating various things, the theory that inodes were insufficient rather than capacity emerged.
inode is the name of a file system used in UNIX-like OS, and it manages file owners, access permissions, size, creation date/time, location, etc.

Remaining inodes can be checked by adding -i to the df command, and when I tried it… bingo.

Source: No space left on device とエラーが出るときの対処法 - Qiita

I found this article.

When I tried it immediately, bingo. It was an inode depletion problem.

$ df -i
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/vda2      1048576 1048576      0  100% /
tmpfs           127541       1 127540    1% /dev/shm

Identifying directories that consume many inodes

I was able to identify directories that consume many inodes with the following script.

echo "==== `pwd` ====" ; for i in `ls -1`;do echo -n "## $i ### " ; echo "(`find ./$i -type f |wc -l`)" ;done | sort -r
Source: inodeとは/inode消費が多いディレクトリの確認

From the root directory, I repeatedly executed the above script and used cd to move to lower hierarchy directories to identify the problematic directory.

wp-file-cache was the cause of inode depletion

When I encountered the No space left on device error, the cause was nearly 880,000 cache files from the WordPress plugin wp-file-cache.

I removed the wp-file-cache plugin as described in the following article.

WordPress キャッシュプラグイン WP File Cache を削除する方法 | CodeNote

Successfully resolved inode depletion problem

After deleting files and checking IUse% with df -i, it dropped from 100% to 15%. I successfully resolved the inode depletion problem, and No space left on device errors stopped occurring.

$ df -i
Filesystem      Inodes  IUsed  IFree IUse% Mounted on
/dev/vda2      1048576 153319 895257   15% /
tmpfs           127541      1 127540    1% /dev/shm

That’s all from the Gemba where I identified inode depletion from No space left on device errors and successfully resolved it.