How to read any process’ /proc/pid/io

There are on Solaris x86-64 some very large files, whose size exceeds 2⁶³, that is, the maximum size representable in an off64_t. This includes the file representing a process's address space in proc (/proc/<pid>/as).

To deal with these files:

1. Don't use fopen, fseek, etc. Don't trust the libc stream routines, which (on the versions of Solaris I tested) mangle badly the "illegal" offsets.
2. Use open64, read.

3. To seek:

   static off64_t lseeku64(int file, uint64_t offset /* eg from pr_argv */)
   {
   #ifndef __sun
     if (offset > 0x7FFFFFFFFFFFFFFFllu) return -1;
   #endif
     return lseek64(file, offset, SEEK_SET);
   }
   

   That is, on Solaris, we know that we can do this cast because of inspection of the OpenSolaris sources, but we should avoid assuming it works on other platforms with psinfo and pr_argv (eg AIX).

   But, pass in your very large offset, and it does all "just work".

Yes, there is a kernel facility: the audit subsystem. The auditd daemon does the logging, and the command auditctl sets up the logging rules. You can log all calls to a specific system alls, with some filtering. If you want to log all commands executed and their arguments, log the execve system call:

auditctl -a exit,always -S execve

To specifically trace the invocation of a specific program, add a filter on the program executable:

auditctl -a exit,always -S execve -F path=/usr/bin/rrdtool

The logs show up in /var/log/audit.log, or wherever your distribution puts them. You need to be root to control the audit subsystem.

Once you're done investigating, use the same command line with -d instead of -a to delete a logging rule, or run auditctl -D to delete all audit rules.

For debugging purposes, replacing the program by a wrapper script gives you more flexibility to log things like the environment, information about the parent process, etc.