Systemd-journald persistent logs do not work with bind mount /var/log

journalctlsystemctlsystemdsystemd-journaldyocto

I am using Yocto to produce a custom image for a small embedded Linux system with SystemD Version 241. The root file system is Read-Only. I am using bind mounts and overlayfs to make the /var/log/journal directory exist on a separate Read/Write partition. I have a problem where systemd-journald gets "Amnesia" and does not remember previous boot logs, even though they are on the persistent Read/Write filesystem. This means journal cannot access or clean older log files from previous boots, even though the log files are present and valid on the filesystem.

Yocto volatile binds

# Setup overlayfs binds for various RW files
VOLATILE_BINDS_append = " \
    /persistent-storage/var/log /var/log\n\
"

The path /var/log exists:

root@me:/var/log# cd /var/log/
root@me:/var/log# ls -lrt
total 9
drwxr-xr-x 2 root root            1024 Jun  3 01:50 nginx
-rw-r--r-- 1 root root            5260 Jun  9 17:56 messages
drwxr-sr-x 5 root systemd-journal 1024 Jun  9 18:00 journal
root@me:/var/log# ls -lrt journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

I have a journald config file fragment in /lib/systemd/journald.conf.d/10-persistent-journal.conf that looks like this:

# By default the maximum use limit (SystemMaxUse) is 10% of the filesystem, and the minimum
# free space (SystemKeepFree) value is 15% - though they are both capped at 4G.
# The journals should be rotated automatically when they reach the SystemMaxFileSize value,
# and the number of journals is controlled by SystemMaxFiles. If you prefer time based
# rotation you can set a MaxFileSec to set the maximum time entries are stored in a single journal.
[Journal]
Storage=persistent
SystemMaxFileSize=128M
SystemMaxFiles=10
SystemMaxUse=256M
SystemKeepFree=256M
SyncIntervalSec=30

The problem is that even though I reboot several times, and journald successfully finds and writes logs to /var/log/journal, it can never find previous logs and has no knowledge about previous boot logs. This means I cannot vacuum previous logs and my partition runs out of space even though journald should maintain 50% of the partition free.

root@me:/# journalctl --list-boots
0 82fef865e29e481aae27bd247c10e591 Tue 2020-06-09 18:00:12 UTC—Tue 2020-06-09
 18:15:23 UTC

Even though:

root@me:/# ls -lrt /var/log/journal/
total 3
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:56 5f6085cd81114e8688cf23b3bb91933e
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 17:57 de59603d1ea24e7582ed7d7ed3ac8fb0
drwxr-sr-x 2 root systemd-journal 1024 Jun  9 18:00 0c34cc794e6c4241a75774bbb8324102

Also, the following commands work:

root@me:/# journalctl -b 0
<information>
root@me:/# journalctl -b 1
<information>

root@me:/# journalctl -b 2
Data from the specified boot (+2) is not available: No such boot ID in journal

I read this post: Can be the journal path on a filesystem other than /?. And I tried the following mount file, however I see exactly the same behavior:

[Unit]
Description=Persistent Journal Storage Bind

[Mount]
What=/anotherfs/journal
Where=/var/log/journal
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

What am I doing wrong and how can I get journald to work with persistent logs on a bind mount system?

Best Answer

Systemd-Journald start at very early stage of the system boot your mount ( /var/log) probably happen after the service is started /var/log. more detail on how you mount your RW partition is needed.

You can try to restart the jounrald service after the system boot.

Storage= Controls where to store journal data. One of "volatile", "persistent", "auto" and "none". If "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed). If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable. "auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes. "none" turns off all storage, all log data received will be dropped. Forwarding to other targets, such as the console, the kernel log buffer, or a syslog socket will still work however. Defaults to "auto" in the default journal namespace, and "persistent" in all others.

Additional info available here: link-1, link-2, link-3, link-3, link-5, link-6, link-7

Related Question