Log Management with the Systemd Journal

Ship's log book Log book of HMS Dolphin, January 1765, Capt. John Byron. Image from Wikimedia Commons.

Systemd adds many features to help administer system and network services running on Linux, and the journal is one of them. The systemd journal stores log messages in a central database which allows for efficient searching of log messages, not just by the message text but also according to metadata about the context of the messages.

Knowing how to find information in log messages can be crucial for many system administration tasks such as troubleshooting, auditing, forensics or monitoring. Therefore, everyone who administers Linux servers should learn how to get the most out of the systemd journal.

Installation and Setup

The systemd journal is installed and enabled by default on most recent GNU/Linux distributions. In some distributions, most notably Debian, the default setup only uses the systemd journal for temporary (in memory) storage, with the traditional syslog used for writing log messages to disk. If persistent storage is not enabled by default, you can enable it with the following commands:

mkdir -p /var/log/journal
journalctl --flush

Configuring Programs to Send Log Messages to the Journal

Some programs and services are systemd-aware, and can send their log messages using the systemd API. In this case, all you have to do is find the relevant configuration option in that program’s configuration files and enable logging to systemd. However, since systemd only runs on Linux, many programs which need to be able to run on other operating systems do not use systemd features. For these programs, there are still many other ways in which you can ensure that the log messages are sent to the systemd journal:

If you run a process as a systemd service, by default its standard output and standard error streams will be sent to the systemd journal. For more information, see the Logging and Standard Input/Output section of the systemd.exec manual page.
The systemd journal daemon listens on the Unix standard syslog interface. Many cross-platform Unix servers can be configured to send their log messages to syslog, and these messages will then be included in the journal.
You can use systemd-cat to forward the output of a program or shell pipeline to the journal.

Sending Log Messages to a Central Log Server

If you have to manage several servers, it is much easier if you can view all the log messages from one place, rather than having to log in to each server separately to see its log messages. Systemd’s solution for this is systemd-journal-remote.

There are two possible modes of operation for remote logging: a “push” mode where the other servers connect to the log server and send messages, and a “pull” mode where the log server connects to the other servers and asks if they have any new messages. The systemd-journal-remote program supports both of these (and also allows importing log data exported with journalctl -o export). For “push” operation, the other servers all install systemd-journal-upload and configure it to connect to the log server. For “pull” operation, the other servers need to have systemd-journal-gatewayd installed, and the log server needs to know which servers it should ask for log messages.

On Debian and Ubuntu, all three programs are included in the “systemd-journal-remote” apt package, which also includes systemd service files and template configuration files. The configuration for systemd-journal-remote goes in /etc/systemd/journal-remote.conf, while that for systemd-journal-upload goes in /etc/systemd/journal-upload.conf. To ensure that the log messages cannot be sniffed or tampered with while travelling over the network, you should configure SSL keys and certificates and choose the “HTTPS” transport.

Usage

The systemd journal stores log messages in a custom binary file format, so you cannot read them with standard Unix text processing tools like grep, less, etc. Instead, the easiest way to access the log messages from the command line is journalctl. Running journalctl without any options will show you all the log messages available. If it detects that its output is going directly to the terminal (rather than redirected to a file or in a shell pipeline), journalctl will pass its output to a pager (usually less) so that you can scroll back and forth and use basic search functionality.

By default, journalctl will display one line per message, with only the most important metadata such as the timestamp. You can use the --output option (-o for short) to get output in a different format. For example, -o verbose will show all metadata, while -o json will give output in a JSON format which is easy for other tools to parse.

Getting the most Recent Messages

If you are used to running tail -f {LOGFILE} to watch a log file in real time, you can get the same functionality with journalctl -f. You can also use journalctl -n {N} to get the same functionality as tail -n {N} {LOGFILE}, although you might find the -e option (pass all log messages to less and scroll to the end) more useful. Instead of selecting the last N messages, you can select messages since a given time with journalctl --since 10:00 for all messages since 10 a.m. or journalctl --since -30min for all messages from the last half hour.

Filtering Log Messages

Probably the most useful filtering option is --unit={NAME}, or -u {NAME} for short. This will give you all messages sent from a given systemd unit (e.g. a system service), plus any messages from systemd itself regarding that unit (e.g. when it was started and stopped, if the process crashed etc.). The --unit option also accepts grep-style patterns. For example, journalctl -u 'network*' will give you messages from network-pre.target, network-online.target, networkd-dispatcher.service etc. You can also use --user to read only the logs of user services (such as a keychain program which caches passwords while the user is logged in) and --system to show only system services. Likewise, --kernel will only show messages from the Linux kernel itself (similar to the dmesg command).

Another useful filtering option is --priority {N} (short form -p {N}). This limits the messages displayed to those which are more important than a given priority level (given either as a number or a name). The naming and numbering of the priority levels is inherited from syslog. Lower numbers mean higher priority; here is the mapping between numbers and names:

0/emerg: Emergency
1/alert: Alert
2/crit: Critical
3/err: Error
4/warning: Warning
5/notice: Notice
6/info: Informational
7/debug: Debug

You can also filter by any of the many metadata fields that the messages include. Use -o verbose to see what metadata fields are available, and then just add FIELDNAME=value to the journalctl command line. For example, this command will select only messages from a particular login session (with session ID 1234):

journalctl _AUDIT_SESSION=1234

To display all messages coming from shell scripts:

journalctl _EXE=/bin/bash

Security Considerations

Confidentiality

Sometimes programs write information to log messages which needs to be kept private. The obvious example of this would be if a log message (or log metadata such as command line arguments) include passwords or other secrets. The ability to see error messages or memory dumps from crashing services can also be helpful for privilege escalation. To guard against this, the default configuration is that unprivileged users can only read journal messages from the services started as part of their user session, and are not able to see messages from the kernel or from system services. The root user can see all messages, as can users that are members of the adm and/or systemd-journal groups.

Integrity

Log messages can be very helpful when investigating security incidents. Many reconnaissance techniques such as port scanning will leave traces in the logs, as do some exploit techniques such as path traversal or credential stuffing. For cases when an authorized user (or a hacker who has taken over that user’s account) uses legitimate privileges to carry out malicious actions, this will also often leave its own log messages. However, none of this is any use if the hacker is able to overwrite or otherwise modify the log files to remove the incriminating messages or add others that would confuse future investigators.

The systemd journal has an optional feature, known as Forward Secure Sealing, which defends against retrospective modification of log messages. Log messages added to the journal are signed with a secret key, and at regular intervals this key is securely deleted and a new key is chosen. A hacker who compromises the system can access the currently valid sealing key and use it to change the logs in the current interval, but has no way of finding out the valid key for previous time periods, meaning that it is impossible to forge a valid signature.

To set up Forward Secure Sealing, run:

journalctl --setup-keys
journalctl --rotate

This will save the signing key in the var/log/journal directory, and write the verification key on the terminal. You should copy the verification key and save it to a different location, where a malicious actor could not access it.

To verify log integrity when investigating an incident, you can use:

journalctl --verify --file {JOURNAL FILE} --verify-key {VERIFICATION KEY}

Availability

Forward Secure Sealing can help protect against undetected modification of log messages, but cannot do anything to stop a hacker or malicious insider simply deleting all your log files. To stop that from happening, you need to ensure that a copy of the log data is available somewhere else, on a system to which the hacker does not have access. Likewise, if your hard drive breaks or your system is otherwise destroyed, you need a backup of your logs somewhere else. The easiest way to do this is by forwarding log messages to one or more remote log servers. For maximum security, these should be in different physical locations, and no one person should have administrative access on every server.

Conclusion

Viewing and analysing log messages is a vital part of every system administrator’s skillset. The systemd journal requires some initial effort to understand its many options and features but, once you have mastered it, the payback in increased efficiency is well worth it.