Linux – strace a short-lived process

eventslinuxlinux-kernelprocess

I'm trying to learn about the behavior of a short-lived process that's created by one of my applications. I know these things about this process:

  1. Part of the name of the process.
  2. The name and PID of the application that will create this process.
  3. Approximately when the process will start (within 5-10 minutes).
  4. After the process has exited, the PID of the process.

Ideally, I would like to:

  1. Have the kernel (or something) notify my script that this process has started.
  2. Run a bunch of tools against the process (iostat, strace, etc).

Is there a way to have the kernel notify me that a certain process has started, so I can take some action on it?

Running something like while true; do ps -ef | grep ${MY_PROCESS_NAME}; done seems bulky and bad. I would like to be able to be notified when it happens, rather than brute-force search for it.

Or, will I just have to run the tools against the parent process and all child processes, then filter through the output later? For example, strace -ff -o ./some.trace -p ${PARENT_PID}.

Best Answer

You might want to look at execsnoop (assuming your kernel was configured with CONFIG_FTRACE, which is usually the case). This is one of many scripts from the Brendan Gregg trace and performance collection. With no args it shows all commands as they start on the system, or you can give it a regexp to watch.

For example, to look for commands that any existing or new zsh might be starting, do:

sudo /opt/perf-tools-master/bin/execsnoop zsh

It shows me this output when I start a new zsh:

Tracing exec()s issued by process name "zsh". Ctrl-C to end.
Instrumenting sys_execve
   PID   PPID ARGS
 21920  21919 /usr/libexec/grepconf.sh -c
 21923  21922 /usr/bin/tty -s
 21922  21919 /usr/bin/tput colors
 21924  21919 /usr/bin/dircolors --sh /etc/DIR_COLORS.256color
 21925  21919 /usr/bin/grep -qi ^COLOR.*none /etc/DIR_COLORS.256color
 21926  21919 /usr/libexec/grepconf.sh -c
 21928  21919 /usr/libexec/grepconf.sh -c
 21930  21919 uname -m
 21932  21919 /bin/grep -q /usr/lib64/qt-3.3/bin
 21934  21933 /usr/bin/id -u

Once you know the full name of the program being run, typically you would replace that file with a script that runs the original program after adding your hooks. If you cannot do that, you can use something like fanotify(7) to have your snooping program intervene before every file open is allowed to complete. Or perhaps inotifywatch would be fast enough for you to attach an strace to the process.

Related Question