How to prevent a process from writing to the systemd journal


I am using a third party .NET Core application (a binary distribution used by a VS Code extension) that unfortunately has diagnostic logging enabled with no apparent way to disable it (I did already report this to the authors). The ideal solution (beside being able to disable it), would be if I could specify to systemd that it should not log anything for that particular program, but I have been unable to find any way to do so. Here is everything I tried so far:

The first thing I tried was to redirect stdout and stderr to /dev/null: dotnet-app > /dev/null 2>&1. This indeed disabled any of the normal output, but the diagnostic logging was still being written to the systemd journal.

I hoped that the application had a command line argument that allowed me to disable the diagnostic logging. It did have a verbosity argument, but after experimenting with, it only seemed to have effect on the normal output, not the diagnostic logging.

By using strace and looking for calls to connect, I found out that the application instead wrote the diagnostic logging directly to /dev/log.

The path /dev/log is a symlink to /run/systemd/journal/dev-log, so to verify my finding, I changed the symlink to point to /dev/null instead. This indeed stopped the diagnostic logging from showing up in the systemd journal.

I was told about LD_PRELOAD and made a library that replaced the standard connect with my own version that returned an error in the case it tried to connect to /dev/log. This worked correctly in my test program, but failed with the .NET Core application, failing with connect ENOENT /tmp/CoreFxPipe_1ddf2df2725f40a68990c92cb4d1ff1e. I experimented with my library, but even if all I did was directly pass the arguments to the standard connect function, it would still fail with the same error.

I then tried using Linux namespaces to make it so that /dev/log would point to /dev/null only for the .NET Core application: unshare --map-root-user --mount sh -c "mount --bind /dev/null /dev/log; dotnet-app $@". This too failed with the same error, even though it again worked for my test program. Even just using unshare --map-root-user --mount dotnet-app "$@" would fail with the error.

Next I tried using gdb to close the file descriptor to /dev/log while the application was running. This worked, but it reopens it after some time has passed. I also tried changing the file descriptor to point to /dev/null, which also worked, but it too was reset to /dev/log after some time.

My last attempt was to write my own UNIX socket that would filter out all written to it by the .NET Core application. That actually worked, but I learned that the PID is send along with what is written to UNIX sockets, so everything passed along to the systemd journal would report coming from the PID of the program backing my UNIX socket.

For now this is solution is acceptable for me, because on my system almost nothing uses /dev/log, but I would welcome a better solution. For example, I read that it was possible to spoof certain things as root for UNIX sockets, but I was unable to find out more about it.

Or if someone might have any insights on why both LD_PRELOAD and unshare might fail for the .NET Core application, while they work fine for a simple C test program that writes to /dev/log?

Best Answer

In short, have your library loaded by LD_PRELOAD override syslog(3) rather than connect(3).

The /dev/log Unix socket is used by the syslog(3) glibc function, which connects to it and writes to it. Overriding connect(3) probably doesn't work because the syslog(3) implementation inside glibc will execute the connect(2) system call rather than the library function, so an LD_PRELOAD hook will not trap the call from within syslog(3).

There's a disconnect between strace, which shows you syscalls, and LD_PRELOAD, which can override library functions (in this case, functions from glibc.) The fact that there's a connect(3) glibc function and also a connect(2) system call also helps with this confusion. (It's possible that using ltrace would have helped here, showing calls to syslog(3) instead.)

You can probably confirm that overriding connect(3) in LD_PRELOAD as you're doing won't work with syslog(3) by having your test program call syslog(3) directly rather than explicitly connecting to /dev/log, which I suspect is how the .NET Core application is behaving.

Hooking into syslog(3) is also potentially more useful, because being at a higher level in the stack, you can use that hook to make decisions such as selectively forwarding some of the messages to syslog. (You can load the syslog function from glibc with dlsym(RTLD_NEXT, "syslog"), and then you can use that function pointer to call syslog(3) for the messages you do want to forward from your hook.)

The approach of replacing /dev/log with a symlink to /dev/null is flawed in that /dev/null will not accept a connect(2) operation (only file operations such as open(2)), so syslog(3) will try to connect and get an error and somehow try to handle it (or maybe return it to the caller), in any case, this might have side effects.

Hopefully using an LD_PRELOAD override of syslog(3) is all you need here.

Related Question