Disk Usage – Find and Remove Large Open but Deleted Files

disk-usagefileslogsopen filesprocess

How does one find large files that have been deleted but are still open in an application? How can one remove such a file, even though a process has it open?

The situation is that we are running a process that is filling up a log file at a terrific rate. I know the reason, and I can fix it. Until then, I would like to rm or empty the log file without shutting down the process.

Simply doing rm output.log removes only references to the file, but it continues to occupy space on disk until the process is terminated. Worse: after rming I now have no way to find where the file is or how big it is! Is there any way to find the file, and possibly empty it, even though it is still open in another process?

I specifically refer to Linux-based operating systems such as Debian or RHEL.

Best Answer

If you can't kill your application, you can truncate instead of deleting the log file to reclaim the space. If the file was not open in append mode (with O_APPEND), then the file will appear as big as before the next time the application writes to it (though with the leading part sparse and looking as if it contained NUL bytes), but the space will have been reclaimed (that does not apply to HFS+ file systems on Apple OS/X that don't support sparse files though).

To truncate it:

: > /path/to/the/file.log

If it was already deleted, on Linux, you can still truncate it by doing:

: > "/proc/$pid/fd/$fd"

Where $pid is the process id of the process that has the file opened, and $fd one file descriptor it has it opened under (which you can check with lsof -p "$pid".

If you don't know the pid, and are looking for deleted files, you can do:

lsof -nP | grep '(deleted)'

lsof -nP +L1, as mentioned by @user75021 is an even better (more reliable and more portable) option (list files that have fewer than 1 link).

Or (on Linux):

find /proc/*/fd -ls | grep  '(deleted)'

Or to find the large ones with zsh:

ls -ld /proc/*/fd/*(-.LM+1l0)

An alternative, if the application is dynamically linked is to attach a debugger to it and make it call close(fd) followed by a new open("the-file", ....).

Related Question