Ubuntu – No space left on device (NTFS)

disk-usagentfsstorageUbuntu

I have a server which stores a lot of data, I keep many nested directories with millions of files in them.

While trying to move one big directory (~700GB) from one disk to another I get an error:

mv /media/storage3/dir /media/storage4/

"No space left on device"

Also, trying to create a small txt file on the disk does not work:

Error opening file '/media/storage4/Untitled Document': No space left on device

I checked many options online, none worked.

df -h output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg7        77G  9.2G   64G  13% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev             32G  4.0K   32G   1% /dev
tmpfs           6.3G  1.4M  6.3G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             32G   76K   32G   1% /run/shm
none            100M   48K  100M   1% /run/user
/dev/sdg6        19G  6.7G   11G  39% /home
/dev/sdb1       440G  278G  140G  67% /var
/dev/sda1       2.8T  2.0T  764G  73% /media/storage1
/dev/sdh1       2.8T  1.6T  1.3T  56% /media/storage2
/dev/sdi1       2.8T  1.7T  1.2T  60% /media/storage3
/dev/sdf1       2.8T  876G  1.9T  32% /media/storage4

df -i output:

Filesystem         Inodes    IUsed      IFree IUse% Mounted on
/dev/sdg7         5079040   514215    4564825   11% /
none              8242449        2    8242447    1% /sys/fs/cgroup
udev              8239750      611    8239139    1% /dev
tmpfs             8242449      696    8241753    1% /run
none              8242449        5    8242444    1% /run/lock
none              8242449        4    8242445    1% /run/shm
none              8242449       28    8242421    1% /run/user
/dev/sdg6         1220608   218613    1001995   18% /home
/dev/sdb1        29310976 12863877   16447099   44% /var
/dev/sda1       858436804 41630853  816805951    5% /media/storage1
/dev/sdh1      1356948436 38728057 1318220379    3% /media/storage2
/dev/sdi1      1217505624 34748869 1182756755    3% /media/storage3
/dev/sdf1      2048962648 36308921 2012653727    2% /media/storage4

Reading online, people said similar problems happen because of access to a huge amount of files and that setting fs.inotify.max_user_watches to a high amount might solve it, I increased it from 8192 to 1000000 but it did not help.

Best Answer

Run the following:

lsof -s | sort -nrk 7 | head

You'll see output like so:

firefox    2997                     j   52rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   50rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j    3rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   39rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   31ur     REG              252,0 10485760    5505182 /some/path

Note; the above is just an example you'll need to use your judgement to find the space usage.

About output:

  • Column 7 is the size of the space in use.
  • Column 4 is the file descriptor.

You can remove the content at the file-descriptor to gain back that space:

cat /dev/null > /proc/2997/fd/50

Obviously this will destroy all data on that file descriptor.

Related Question