Linux – System-wide monitoring of calls to a library function

linuxltracesystemtap

I have a modern Linux desktop with lots of processes running concurrently. One of those processes, and I don't know which one, calls a function some_func from a popular dynamic library some_lib (think libc or libx11, so a lot of processes use it), and I want to know what process does that (and ideally, have a stack trace of each invocation).

How do I determine which process makes a call to some_lib?

Options that I've considered thus far:

  1. Use ltrace or latrace: Having an ltrace-style detailed list
    of what process called the function I'm interested in with what
    arguments would be perfect, but ltrace only works with an
    individual process or process groups. I cannot just type ltrace -e
    some_func@some_lib -fp 1     and see all uses system-wide.
  2. Find what processes use my library with lsof, then proceed with step 1: That would be very cumbersome, since there are too many processes using the same library, but not calling said function.
  3. grep -r some_func /usr, then see if there are only a couple of binaries capable of calling the function, and work my way from there. Although that could work in some limited amounts of cases, this is by no means a general solution, and wouldn't work if e.g. some_func is ubiquitous in various binaries but is seldom called.
  4. Use the kernel audit system. If I was tracing a system call, I could type auditctl -S some_syscall ... and that would do the trick in logging system-wide invocations. However, auditctl does not seem to be capable of doing the same level of granularity with library functions.
  5. Finally, I could rebuild the library, adding a new line to the function I'm interested in that would log all its invocations. While this would be guaranteed to solve my issue, that solution would be cumbersome and require modifying/recompiling the library and at least 2 reboots to roll out the instrumented library and to roll it back after finding the culprit.

Is there an easier way?

(I want to point out that this is meant to be a general question, and am mostly interested in general solutions that would Just Work.)

I found a nice comparison article mentioning a few more tracing facilities I wasn't aware of, which may be worth exploring.

Best Answer

SystemTap with debuginfo can trace function calls in libraries; on a Centos 7 system:

$ sudo stap -L 'process("/lib64/libglib*").function("*strndup*")'
process("/usr/lib64/libglib-2.0.so.0.5000.3").function("g_strndup")
$

And this can be used as a probe point that prints backtraces or whatever you want that can be written with SystemTap:

probe begin {
    printf("ok\n")
}
probe process("/usr/lib64/libglib-2.0.so.0.5000.3").function("g_strndup") {
    /* printf("%s[%d]\n", execname(), pid()) */
    print_usyms(ubacktrace())
}

saved as probelibraryfunc.stp this can be run via

$ sudo stap probelibraryfunc.stp

though may produce crazy amounts of output if the call is common...

Related Question