Linux – How to not log some messages in systemd journal

arch linuxsystemdsystemd-journald

Using Archlinux aarch64, I have an error message that keeps happening, which apparently cannot be solved, at least not simply (see this question). Since it occurs very frequently I am concerned about log size, and the storage being a micro-sd card, about its premature wear.

Is there a way to not logs these messages? I see in journald doc it's possible to limit by rate or max size, but I don't see anything for what I want. Ideally it would not filter out the unit completely, since the message is from the kernel.

The error message is always the same, so maybe there's another trick?

Best Answer

From reading man systemd-journald and man journald.conf, I don't see an option for discarding incoming logics, which is possible with or logging daemons like rsyslog.

The workarounds I'm aware of are:

Logging into memory instead of the filesystem-- Set Storage=volatile in journald.conf. The volatility is an obvious tradeoff.
The ability to limit the maximum space consumed by the logs with SystemMaxUse= and RunTimeMaxUse=.

A more extreme workaround is to use the options above to limit the resource consumption by the systemd journal, while also forwarding the logs on to another logging daemon which has a filtering feature.

See man journald.conf for ForwardToSyslog= and other options.

Related Solutions

Pulling log messages for a particular log in systemd journal

Yes, it is possible, but you passed the wrong switch to journalctl.

According to journalctl(1) man page:

To read messages with a given syslog identifier (say, "foo"), issue journalctl -t foo or journalctl SYSLOG_IDENTIFIER=foo;
To read messages with a given syslog facility, issue journalctl SYSLOG_FACILITY=1 (note that facilities are stored and matched using their numeric values).

More generally, the syslog identifier and facility are stored in the journal as separate fields (SYSLOG_IDENTIFIER and SYSLOG_FACILITY). If you ever need to access the journal from, say, the C API, you will have to add matches on these fields directly.

The journalctl -u switch is used to add a match on the name of a systemd unit which owned the process which has generated the message. So this is the wrong switch to use.

Systemd – Why is Systemd Journal Not Persistent Across Reboots?

The reason for this behavior is that the machine identifier in /etc/machine-id changes at every reboot. This starts a new logging directory under /var/log/journal. Old logs can be viewed with the following command:

journalctl --merge

I'm still looking into the cause of the changing machine-id. Linode support is aware of the problem. I will update this answer when I know more.

UPDATE -- The root cause of the problem is simply that Linode zeroed out the contents of /etc/machine-id from their filesystem images. The result is the following chain of events:

The kernel loads and mounts the root filesystem read-only
systemd, run from the initial ramdisk, tries to read /etc/machine-id from the root filesystem (the file exists but has zero contents)
systemd cannot read the machine identifier, but can also not write a new one since the root filesystem is mounted read-only
systemd mounts tmpfs on /etc/machine-id (Yes, apparently you can mount a filesystem onto a file)
systemd invokes systemd-machine-id-setup which generates a random machine-id and stores it in the now-volatile /etc/machine-id
The system boots with a volatile machine identifier

You can check if your system has a volatile, rather than a permanent machine-id by looking at the output of mount:

$ mount | grep machine-id
tmpfs on /etc/machine-id type tmpfs (ro,mode=755)

The problem is easy to fix: simply write a persistent machine-id to the real /etc/machine-id. This is easier said than done, however, because you cannot unmount tmpfs from /etc/machine-id on a running system. These are the steps I took to fix it on Linode:

cp /etc/machine-id /etc/machine-id.copy, then poweroff the system
In the Linode Manager, go to the tab Rescue and boot into rescue mode
Access the system via the Lish console
Mount the root filesystem: mount /dev/xvda /mnt
Move the copy created in step 1 to the real machine-id: mv /etc/machine-id.copy /etc/machine-id
Reboot

Such are the consequences of a missing machine-id at boot. I hope this will help a random passer-by in the future.

Best Answer

Related Solutions

Pulling log messages for a particular log in systemd journal

Systemd – Why is Systemd Journal Not Persistent Across Reboots?

Related Question