Linux – How to Access a Deleted Open File (Output of a Running Crontab Task)

If a file has been deleted but is still open, that means the file still exists in the filesystem (it has an inode) but has a hard link count of 0. Since there is no link to the file, you cannot open it by name. There is no facility to open a file by inode either.

There is no way to discover the file through its filesystem, and especially no way to look for the file in the directory where it last was. The directory entry is gone. All that remains is the file itself. You can get to the file with a filesystem debugger, but that requires root permissions and is hard to use and error-prone.

Linux exposes open files through special symbolic links under /proc. These links are called /proc/12345/fd/42 where 12345 is the PID of a process and 42 is the number of a file descriptor in that process. A program running as the same user as that process can access the file (the read/write/execute permissions are the same you had as when the file was deleted).

The name under which the file was opened is still visible in the target of the symbolic link: if the file was /var/log/apache/foo.log, then the target of the link is /var/log/apache/foo.log (deleted). (If the file was renamed after it was opened, the target of the symlink may reflect the renaming.)

Thus you can recover the content of an open deleted file given the PID of a process that has it open and the descriptor that it's opened on like this:

recover_open_deleted_file () {
  old_name=$(readlink "$1")
  case "$old_name" in
    *' (deleted)')
      old_name=${old_name%' (deleted)'}
      if [ -e "$old_name" ]; then
        new_name=$(TMPDIR=${old_name%/*} mktemp)
        echo "$oldname has been replaced, recovering content to $new_name"
      else
        new_name="$old_name"
      fi
      cat <"$1" >"$new_name";;
    *) echo "File is not deleted, doing nothing";;
  esac
}
recover_open_deleted_file "/proc/$pid/fd/$fd"

If you only know the process ID but not the descriptor, you can recover all files with

for x in /proc/$pid/fd/*; do
  recover_open_deleted_file "$x"
done

If you don't know the process ID either, you can search among all processes:

for x in /proc/[1-9]*/fd/*; do
  case $(readlink "$x") in
    /var/log/apache/*) recover_open_deleted_file "$x";;
  esac
done

You can also obtain this list by parsing the output of lsof, but it isn't simpler nor more reliable nor more portable (this is Linux-specific anyhow).

It could only be in memory and not recoverable, in which case you'd have to try to recover it from the filesystem using one of those filesystem recovery tools (or from memory, maybe). However!

$ cat hamlet.c
#include <unistd.h>
int main(void) { while (1) { sleep(9999); } }
$ gcc -o hamlet hamlet.c
$ md5sum hamlet
30558ea86c0eb864e25f5411f2480129  hamlet
$ ./hamlet &
[1] 2137
$ rm hamlet
$ cat /proc/2137/exe > newhamlet
$ md5sum newhamlet 
30558ea86c0eb864e25f5411f2480129  newhamlet
$ 

With interpreted programs, obtaining the script file may be somewhere between tricky and impossible, as /proc/$$/exe will point to perl or whatever, and the input file may already have been closed:

$ echo sleep 9999 > x
$ perl x &
[1] 16439
$ rm x
$ readlink /proc/16439/exe
/usr/bin/perl
$ ls /proc/16439/fd
0  1  2

Only the standard file descriptors are open, so x is already gone (though may for some time still exist on the filesystem, and who knows what the interpreter has in memory).