space ≠ space
Has it ever happened to you that it seemed as if there was still space on your hard disk, but somehow there wasn’t?
$:/home# touch blub
touch: cannot touch 'blub': No space left on device$:/home# df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 1.3M 3.2G 1% /run
/dev/mapper/vg0-root 15G 2.7G 11G 20% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/md0 232M 151M 70M 69% /boot
/dev/mapper/vg0-tmp 9.4G 37M 8.8G 1% /tmp
/dev/mapper/vg0-var 47G 37G 8.3G 82% /var
/dev/mapper/vg0-home 632G 348G 252G 58% /home
As it turns out there’s not only a storage space limit but but also an address space limit. Files are references by Inodes. Your file system only has space for a certain amount of Indodes. In my case this limit was 42 Mio. of which all where used.
$:/home# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 4057004 506 4056498 1% /dev
tmpfs 4064966 906 4064060 1% /run
/dev/mapper/vg0-root 938400 154854 783546 17% /
tmpfs 4064966 1 4064965 1% /dev/shm
tmpfs 4064966 3 4064963 1% /run/lock
tmpfs 4064966 18 4064948 1% /sys/fs/cgroup
/dev/md0 62248 318 61930 1% /boot
/dev/mapper/vg0-tmp 625856 19 625837 1% /tmp
/dev/mapper/vg0-var 3129344 534364 2594980 18% /var
/dev/mapper/vg0-home 42131456 42131456 0 100% /homeThis means, I have a lot of files on my disk. Let’s find where they are by running the following command (source: https://unix.stackexchange.com/questions/122854/find-the-top-50-directories-containing-the-most-files-directories-in-their-first):
du --inodes -xS | sort -rh | sed -n \
'1,50{/^.\{71\}/s/^\(.\{30\}\).*\(.\{37\}\)$/\1...\2/;p}'I wasn’t patient enough to fully run it because I in the meantime found out that I have about 35 Mio. files which I really need. So I need to create a new filesystem and specify the Inode count manually(mkfs.ext4 -i ).
