Linux – filter out certain processes and/or pids in ftrace

ftracelinuxprocesssystem-callstracing

I'm not sure if I'm thinking about this the right way (and please correct me if I'm wrong), but the following is my understanding of ftrace.

In /sys/kernel/debug/tracing, there are the following files:

set_ftrace_filter

which will only trace the functions listed inside,

set_ftrace_notrace

which will only trace the functions NOT listed inside, and

set_ftrace_pid

which will only trace the processes with the pid inside.

My question is: is there a way to configure it so that ftrace will only trace processes that DO NOT have a certain pid (or process name)?

Analogy:

set_ftrace_filter : set_ftrace_notrace :: set_ftrace_pid : x

Does x exist, and if so, how do I use it?

For instance, if I wanted to trace all processes except the one with pid 48, is there some way to put something meaning not 48 into set_ftrace_pid?

I've been reading the documentation and searching the web, but I can't find either the way to achieve this or whether this is possible.

Why I'm doing this: I have a program that's tracing kernel-level system calls, but I want to write the program's pid (and the pids of its children, if necessary later) to a filter so that they aren't included with the trace data. When reading the trace, I could check the pid as I read each trace record and decide whether to use that record or not, but I would prefer not to add this overhead for every record that is read if there's a way to avoid it.

Thank you for your time!

Best Answer

I figured out how to do what I was describing, but it was a bit counter-intuitive, so I'm posting the answer here for people who might hit this page when searching (tl:dr; at bottom). As far as I know, there is no blanket way to just filter out processes with a certain PID from ftrace as easily as it is to tell it to ONLY consider processes with a certain PID, but in my case, I only care about raw system calls (sys_enter) and I found out how to eliminate records with certain PIDs from being included for those and this is how:

The ftrace directory is:

/sys/kernel/debug/tracing/

Inside, there is a directory called "events." From here, you can see all the things that ftrace can trace, but for my case, I go into "raw_syscalls."

Within raw_syscalls," the two subdirectories are sys_enter and sys_exit.

Within sys_enter (and sys_exit, for that matter), there are the following files:

enable

filter

format

id

trigger

"filter" is the one we care most about right now, but format has useful information regarding the fields of an entry produced by ftrace when sys_enter is enabled:

name: sys_enter
ID: 17
format:
    field:unsigned short common_type;   offset:0;   size:2; signed:0;
    field:unsigned char common_flags;   offset:2;   size:1; signed:0;
    field:unsigned char common_preempt_count;   offset:3;   size:1; signed:0;
    field:int common_pid;   offset:4;   size:4; signed:1;

    field:long id;  offset:8;   size:8; signed:1;
    field:unsigned long args[6];    offset:16;  size:48;    signed:0;

Here, we care about common_pid.

If you want your trace to omit records from a process with PID n, you would edit

/sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/filter

To read:

common_pid != n

If the program you're trying to ignore while tracing has multiple threads or multiple processes, you just use the && operator. Say you want to omit processes with PIDs n, o, and p, you would edit the file so that it reads:

common_pid != n && common_pid != o && common_pid != p

To clear a filter, you just write "0" to the file:

echo "0" > /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/filter

...would do the trick.

enable has to contain "1" for the event you're tracing as well as tracing_on in the ftrace directory. Writing in 0 turns tracing of that event (or all tracing in the case of tracing_on) off.

Writing to these files requires root permissions.

That's about all I can think of. Thanks to anyone who read/voted on this and I hope my answer helps someone. If anyone knows a way that makes the way I did it look stupid, feel free to call me out.

tl;dr: to filter out records from process 48, write:

common_pid != 48

...to

/sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/filter

Filter multiple PIDs (eg. 48, 49, 53, 58) by writing this instead:

common_pid != 48 && common_pid != 49 && common_pid != 53 && common_pid !=58

Replace "events/raw_syscalls/sys_enter" with your desired event and replace my numbers with whatever PIDs you want to ignore.

Related Question