systemd Journal Filling /var on a WHM Server: Vacuum Once or Set SystemMaxUse?

A WHM server warns that /var is almost full. Someone runs a quick du, finds several gigabytes sitting in /var/log/journal, runs a vacuum command, and the alert clears. A few weeks later it is back. So is vacuuming the fix, or only the cleanup? Vacuum is a one-time deletion. SystemMaxUse is a rule journald follows by itself. If the problem keeps coming back, you need the rule. To see why, it helps to look at how journald writes, rotates, and deletes its own files. Once that is clear, the choice between the two approaches mostly makes itself.

Why a full /var hurts more on a WHM server

On a typical cPanel and WHM install, /var holds more than logs. MySQL or MariaDB keeps its data under /var/lib/mysql, and Exim keeps its mail queue under /var/spool/exim. When the filesystem holding /var runs out of space, the database can stop accepting writes and mail can stop queueing. To the customer it looks like a broken site or missing email, not a log problem. That is why a support engineer usually checks disk space early when several unrelated services fail at the same time. It is also why the journal deserves a fixed cap. It is one of the few things on /var that grows without anyone touching it. Start by measuring. These commands are read-only:

BASH

# How much space journald reports using (active + archived files)
journalctl --disk-usage

# Which filesystem holds /var, and how full it is
df -h /var

# Journal files per machine ID directory, with sizes
ls -lh /var/log/journal/*/

# Effective journald configuration, including drop-in files
systemd-analyze cat-config systemd/journald.conf

If /var/log/journal does not exist, journald is probably keeping logs only in memory under /run/log/journal. Then the journal is not what is filling /var. Whether persistent storage is on by default depends on the distribution, so check your own server instead of assuming.

How journald writes and rotates its files

systemd-journald is a daemon that collects log messages from several sources. Services started by systemd have their stdout and stderr connected to journald through a socket under /run/systemd/journal/. Programs that use the classic syslog API write to /dev/log, which journald also listens on. Kernel messages come in through /dev/kmsg. Programs can also use journald's native socket directly. All of that goes into one active file per journal, usually named system.journal, plus per-user files if user journals are kept. The file is a binary, indexed format. journald memory-maps it and appends entries, along with hash tables and entry arrays so journalctl can filter by unit, priority, or time without reading everything. The active file does not grow forever. journald rotates it when one of these happens:

The file reaches SystemMaxFileSize.
It reaches the age set by MaxFileSec, if that is set.
Someone runs journalctl --rotate.
journald finds the file damaged or not closed cleanly.

On rotation, journald renames the active file to an archived name. That name looks roughly like system@ followed by a sequence ID, the first sequence number, and the first timestamp. Then journald opens a fresh system.journal. Files that were not closed cleanly get a trailing ~ in their name instead. After rotating, journald checks its size rules. If the total is over the limit, it deletes the oldest archived files until it is back under. This step is the important one. Automatic cleanup is tied to rotation, so it only works as well as the limits it reads.

What vacuum does, and what it leaves alone

journalctl --vacuum-size, --vacuum-time, and --vacuum-files run the same kind of deletion on demand. journalctl walks the journal directories, sorts archived files from oldest to newest, and removes them until the target is met. With --vacuum-size=500M, it deletes until the archived files total 500M or less. With --vacuum-time=2weeks, it removes archived files whose newest entry is older than two weeks. Two details explain most of the confusion here. First, vacuum only touches archived files. The active system.journal is never deleted, because journald is still writing to it. If most of the space sits in a large active file, a vacuum seems to do almost nothing. Running journalctl --rotate first archives the current file so the vacuum can include it. Second, vacuum is a single action with no memory. It does not write anything to journald.conf. The next time journald rotates, it uses whatever limits were already configured. If those limits let the journal reach several gigabytes, it will get there again. The same goes, more painfully, for deleting journal files with rm. journald may still have those files open and memory-mapped. In that case the directory entry disappears but the space stays allocated until the daemon closes them. You get a disk that looks full even though the files seem to be gone. Using journalctl's own vacuum avoids this, because it skips the active files and removes archived ones properly. Keep in mind that vacuum deletion is permanent. If you need logs for an ongoing investigation, export or copy what you need before trimming.

What SystemMaxUse and its siblings actually control

The lasting fix is in journald.conf, under the [Journal] section. The persistent-storage settings that matter here are:

SystemMaxUse: the most disk space all persistent journal files together may use.
SystemKeepFree: how much free space journald must leave on the filesystem for everything else.
SystemMaxFileSize: how large one journal file may grow before rotation.
MaxRetentionSec: an optional age limit, so entries older than this are removed even if there is space.

journald enforces whichever of SystemMaxUse and SystemKeepFree is stricter at the moment. When neither is set, the documented defaults are a percentage of the filesystem size: 10% for SystemMaxUse and 15% for SystemKeepFree, each capped at 4G. Check man journald.conf on your server, because these defaults can change between systemd versions. Those defaults explain why the journal can still take up a lot of room. Ten percent of a large root filesystem is a lot of logs, and the 4G cap may still be too much if /var shares space with a busy database. The other common cause is that someone raised the limits earlier, maybe to keep logs longer while debugging, and never set them back. That is why it is worth reading systemd-analyze cat-config before changing anything. A drop-in file may already be overriding the main config. When journald applies the limits also matters. It checks them mainly at rotation and at startup, not on every write. So usage can sit a bit above the cap until the next rotation. A smaller SystemMaxFileSize means more frequent rotations, so the cap is enforced more closely. The trade-off is more files in the directory. If you want to see where that space goes on lightly used systems, the article on minimizing RAM and disk used by systemd-journald covers the more aggressive end of the settings.

Applying a cap that survives

Putting the limits in a drop-in file is safer than editing journald.conf directly. Package updates are less likely to overwrite it, and it is easy to find later. The values below are an illustrative example for a server where /var has limited room. Size them for your own partition and for how long you need to keep logs.

INI

# /etc/systemd/journald.conf.d/size.conf
# Illustrative example values; adjust to your /var size
[Journal]
SystemMaxUse=1G
SystemKeepFree=2G
SystemMaxFileSize=128M

Then load the new limits and do one cleanup to get down to them right away. The vacuum step deletes old logs permanently, so copy anything you still need first.

BASH

# Create the drop-in directory if it does not exist, then add size.conf
sudo mkdir -p /etc/systemd/journald.conf.d

# Restart journald so it reads the new limits
sudo systemctl restart systemd-journald

# Archive the current active file so vacuum can include it
sudo journalctl --rotate

# One-time trim down to the new cap (alternative: --vacuum-time=2weeks)
sudo journalctl --vacuum-size=1G

# Confirm the result
journalctl --disk-usage

Restarting systemd-journald is normally safe, and services keep logging through it. The order here is deliberate. The config makes the limit permanent, and the vacuum handles the backlog that built up before the limit existed. Running only the vacuum gives you the same alert again later. Setting only the limit works too, but the space comes back gradually as rotations happen, not right away. If /var/log/journal holds several machine ID directories, the server may have been cloned or migrated, and the extra directories may belong to the old system. Check what they are before removing anything, and do not assume journald's automatic cleanup covers them. A cap stops the journal from taking over the disk, but it does not explain why it was growing so fast. If journalctl --disk-usage reaches the new limit quickly, look at what is writing the most, such as a unit stuck in a restart loop or a service logging at debug level. Then fix the source rather than raising the cap. In practice: vacuum when you need space back right now, and set SystemMaxUse and SystemKeepFree so you do not have to do it again. Check the effective config first, size the limits against the partition that really holds /var, and watch the journal's growth for a while afterward so you notice early if a noisy service is still filling it.