Ubuntu – Where do log messages go with journald and rsyslog

loggingrsyslogserversyslogsystemd-journald

On Ubuntu 18.04 both journald and rsyslog are installed. Both serve the same purpose of collecting log messages and storing them. So we have two programs doing the same work here.

What I could find out is that journald only saves log messages in its own journal files which can be queried with the journalctl command. No readable files are written. Also, journal will always collect log output from all the services that were started by systemd, which should always be all of them because ultimately systemd starts all processes even from older schemes like init scripts.

Rsyslog also receives log messages, through some socket that works like syslog has worked the past 50 years or so. Applications have to send their messages to that socket and rsyslog will receive them. Not sure how journald actually receives anything. Then rsyslog forwards these received messages into different readable files as per its configuration.

In Ubuntu, both are installed but do not seem to be connected. At least rsyslog does not load the "imjournal" module with which it could "read" log messages from journald. I don't understand how rsyslog (actively) reads anything, I thought it only (passively) receives messages. But it's not using that anyway.

The file /var/log/syslog contains largely the same content as shown by journalctl, just the output from sudo is missing in the file. But I haven't compared it all.

So how does this work? Who sends log messages to where and who receives them and where are they forwarded to? If both destinations are filled, does every application have to send messages to both syslog and journald separately? If they are interconnected, why are no messages duplicated? Are they filtered out again? Which location is more complete and where should I look if I want the complete picture?

If any of this has changed in newer versions of Ubuntu, I'd also be interested in that, as I'm going to switch an older server to 20.04 next year.

Edit: Here are some more resources on the topic:

All of this information leaves me a bit confused. It seems like journal receives all syslog data from the system call in the first place (my /dev/log points to /run/systemd/journal/dev-log), and rsyslog fetches it from journal through a socket that looks like the socket that is had used before there was journal (possibly /run/systemd/journal/syslog, implicitly used if it sees systemd around). Then rsyslog can do all its advanced log processing (including writing the legacy /var/log files). If I don't need any of that magic, I might just disable rsyslog. But I'm not sure about that at all!

Best Answer

Message storage locations

journald message storage

When it's time to clean up older messages in journald (I do it monthly) you can see the files where messages are stored:

Deleted archived journal /var/log/journal/1ff17e6df1874fb3b2a75e669fa978f1/system@00059368465fc63c-5ca08f36fa6e6f04.journal~ (32.0M).
Deleted archived journal /var/log/journal/1ff17e6df1874fb3b2a75e669fa978f1/user-1000@0005936848a6d1ec-316ff74be646031e.journal~ (8.0M).

As you can see the file names are very long and cryptic.

syslog message storage

syslog messages are kept in a more traditional filename format:

$ ll /var/log/syslog*
-rw-r----- 1 syslog adm   2161 Nov 29 04:47 /var/log/syslog
-rw-r----- 1 syslog adm 159700 Nov 29 04:32 /var/log/syslog.1
-rw-r----- 1 syslog adm  24466 Nov 28 04:27 /var/log/syslog.2.gz
-rw-r----- 1 syslog adm  23489 Nov 27 04:30 /var/log/syslog.3.gz
-rw-r----- 1 syslog adm  28087 Nov 26 04:28 /var/log/syslog.4.gz
-rw-r----- 1 syslog adm  28300 Nov 25 04:30 /var/log/syslog.5.gz
-rw-r----- 1 syslog adm  27445 Nov 24 05:36 /var/log/syslog.6.gz
-rw-r----- 1 syslog adm  27460 Nov 23 08:25 /var/log/syslog.7.gz

The filenames ending in .gz have compressed data to save space on disk.

Messages recorded are not the same

A quick test reveals the two message logging systems aren't exact duplicates. We'll put a message into the system with logger command and then search for it and display the five lines before it.

$ logger $0 "ygoe to two logs?"



$ cat /var/log/syslog | grep ygoe -a5
Nov 27 20:53:35 alien upowerd[2032]: message repeated 3 times: [ (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:101A.017F/power_supply/hidpp_battery_63]
Nov 27 20:54:23 alien upowerd[2032]: (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:2010.0180/power_supply/hidpp_battery_64
Nov 27 20:59:25 alien wpa_supplicant[1591]: wlp60s0: WPA: Group rekeying completed with ae:20:2e:cc:94:50 [GTK=CCMP]
Nov 27 21:00:02 alien CRON[24890]: (root) CMD (/usr/bin/updatedb)
Nov 27 21:07:48 alien upowerd[2032]: (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:2010.0180/power_supply/hidpp_battery_64
Nov 27 21:08:22 alien rick: bash ygoe to two logs?



$ journalctl -xe | grep ygoe -a5
Nov 27 20:59:25 alien wpa_supplicant[1591]: wlp60s0: WPA: Group rekeying completed with ae:20:2e:cc:94:50 [GTK=CCMP]
Nov 27 21:00:01 alien CRON[24881]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 27 21:00:01 alien CRON[24890]: (root) CMD (/usr/bin/updatedb)
Nov 27 21:00:04 alien CRON[24881]: pam_unix(cron:session): session closed for user root
Nov 27 21:07:48 alien upowerd[2032]: (upowerd:2032): UPower-Linux-WARNING **: treating change event as add on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9:1.2/0003:046D:C52B.017E/0003:046D:2010.0180/power_supply/hidpp_battery_64
Nov 27 21:08:22 alien rick[8000]: bash ygoe to two logs?

As you can see journalctl provides more details than syslog. Additionally journalctl:

There was debate among developers about duplicating journald and syslog a few years ago however, I couldn't find the link just now.

Related Question