I was running a Python process which creates a huge number of files under a single directory (I should have been smarter and bucketed them into multiple directories, but that's another story).
After a while, I noticed that I couldn't go further and the script started giving me an error:
IOError: [Errno 28] No space left on device: /scr1/data/somefile_00023532.txt
Upon researching I've found that the most common reasons are (1) the partition really does not have space left and (2) all inodes are used up. However, df shows
$ df -h
...
/dev/sdb1 2.7T 1.2T 1.4T 46% /scr1
...
(/dev/sdb1 is the partition in question) and
$ df -i
...
/dev/sdb1 183148544 17725595 165422949 10% /scr1
...
so both space and inodes are indeed still available.
There may be too many files:
$ ls /scr1/data | wc
6468500 6468500 349747747
but ext4 should be able to handle this.
What could be causing Linux to think the space is used up?
UPDATE 1
It appears with problem happens with a specific file name. For example,
$ touch /scr1/data/somefilewithproblem.txt
touch: cannot touch ‘/scr1/data/somefilewithproblem.txt‘: No space left on device
but other files with the same file name pattern (I'm using something benign, like a hash in hex, numeric ID number, etc.) do not have the same issue.
UPDATE 2
DUH!! It looks like the filesystem was corrupt somehow, and after running fsck.ext4 on the partition in question, the issue went away.
Thank you very much for those who offered me help!
Best Answer
"No space left on device" can be a very missleading error. It can pop up on all sorts of write conditions other than having no disk space available.
Does that file already exist and is owned by someone else?
Do you have quotas enabled on this filesystem?
If you try to create another file in that filesystem manually (ie
touch /src1/data/testfile) do you get the same error, or does it create the file?Are you able to try creating a file as root?