PSA - Systemd Journal - persistence settings and race condition between zfs-mount.service and systemd-journald.service

Many of the Linux Root-on-ZFS recipes make use of minimal network installers such as debootstrap. For distros that use Systemd, there are a couple of things related to the Systemd journal that can cause problems with logging.

The journal is not persistent by default

By default, systemd-journald stores the journal as an in-memory ring buffer under /run/log/journal/
This means that journal entries are lost on reboot, which is not useful if you need to debug a crashing machine. Most distro installers set specific journal options to ensure the journal is persistent and/or forwards it’s entries to syslog.
If you have used a minimal installer stuch as debootstrap, you may need to set these options yourself.

To check if your journal is persistent, see if /var/log/journal/ is present and has files under it.

root@server /root# ls -l /var/log/journal
total 36
drwxr-sr-x+ 2 root systemd-journal 4 Nov 21 14:36 df459cea02ed4e92981dfc201fded7ec/

If it is not present, run the following:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

This is all you need to enable a persistent journal from next boot. To activate it now, you can use

journalctl --flush
systemctl restart systemd-journald

It also might be a good idea to set some options for the persistent journal (man journald.conf for a list)

mkdir -p /etc/systemd/journald.conf.d/
cat > /etc/systemd/journald.conf.d/99-persistent-journal.conf << EOF
[Journal]
Storage=persistent
Compress=yes
MaxFileSec=1day
ForwardToSyslog=no
EOF

Race condition between zfs-mount.service and systemd-journald.service

With your persistent journal being stored on a zfs filesystem, there is now a race condition on startup beween zfs-mount.service, which will mount your /var/log directory, and systemd-journald.service, which will try to write to it before it’s been mounted. This can cause systemd-journald to fall back to using an in-memory journal, or result in unpredictable loging behaviour (ask me how I know :confused:)

The fix is to create a dependency for systemd-journald.service on zfs-mount.service:

mkdir -p /etc/systemd/system/systemd-journald.service.d/
cat > /etc/systemd/system/systemd-journald.service.d/zfs.conf << 'EOF'
[Unit]
After=zfs-mount.service
Requires=zfs-mount.service
EOF

After this you should have a persistent journal that is correctly stored on your zfs dataset.

3 Likes