I have a server which stores a lot of data, I keep many nested directories with millions of files in them.
While trying to move one big directory (~700GB) from one disk to another I get an error:
mv /media/storage3/dir /media/storage4/
"No space left on device"
Also, trying to create a small txt file on the disk does not work:
Error opening file '/media/storage4/Untitled Document': No space left on device
I checked many options online, none worked.
df -h output:
Filesystem Size Used Avail Use% Mounted on
/dev/sdg7 77G 9.2G 64G 13% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 32G 4.0K 32G 1% /dev
tmpfs 6.3G 1.4M 6.3G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 32G 76K 32G 1% /run/shm
none 100M 48K 100M 1% /run/user
/dev/sdg6 19G 6.7G 11G 39% /home
/dev/sdb1 440G 278G 140G 67% /var
/dev/sda1 2.8T 2.0T 764G 73% /media/storage1
/dev/sdh1 2.8T 1.6T 1.3T 56% /media/storage2
/dev/sdi1 2.8T 1.7T 1.2T 60% /media/storage3
/dev/sdf1 2.8T 876G 1.9T 32% /media/storage4
df -i output:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdg7 5079040 514215 4564825 11% /
none 8242449 2 8242447 1% /sys/fs/cgroup
udev 8239750 611 8239139 1% /dev
tmpfs 8242449 696 8241753 1% /run
none 8242449 5 8242444 1% /run/lock
none 8242449 4 8242445 1% /run/shm
none 8242449 28 8242421 1% /run/user
/dev/sdg6 1220608 218613 1001995 18% /home
/dev/sdb1 29310976 12863877 16447099 44% /var
/dev/sda1 858436804 41630853 816805951 5% /media/storage1
/dev/sdh1 1356948436 38728057 1318220379 3% /media/storage2
/dev/sdi1 1217505624 34748869 1182756755 3% /media/storage3
/dev/sdf1 2048962648 36308921 2012653727 2% /media/storage4
Reading online, people said similar problems happen because of access to a huge amount of files and that setting fs.inotify.max_user_watches to a high amount might solve it, I increased it from 8192 to 1000000 but it did not help.
Best Answer
Run the following:
lsof -s | sort -nrk 7 | head
You'll see output like so:
Note; the above is just an example you'll need to use your judgement to find the space usage.
About output:
You can remove the content at the file-descriptor to gain back that space:
cat /dev/null > /proc/2997/fd/50
Obviously this will destroy all data on that file descriptor.