Moved to a Minimal Cloud Image and journald Now Keeps Logs in RAM

You move a service onto a minimal cloud image. A few days later the VPS reboots after a kernel update, and journalctl -b -1 returns nothing about the previous boot. Or free shows the 'shared' column creeping up while no process looks responsible. On many minimal images both symptoms have the same cause. journald is running in volatile mode and keeps the whole journal in a tmpfs under /run/log/journal, which lives in RAM, instead of on disk. This catches people because nothing is broken. journalctl works, systemctl status shows recent log lines, and the service runs. The difference only shows up when you need logs from before a reboot, or when a chatty application runs on a VPS with little memory.

How journald decides where to write

journald, the systemd logging daemon, collects messages from several sources. Applications send them through the syslog socket, systemd's native journal socket, or their stdout and stderr when systemd starts them as a service. The kernel sends its messages through /dev/kmsg. journald adds metadata such as the unit name, PID, and timestamps, then appends each entry to a structured journal file. Where that file lives is controlled by the Storage= option in /etc/systemd/journald.conf and its drop-in directory. It accepts four values:

volatile keeps logs only under /run/log/journal.
persistent keeps logs under /var/log/journal and creates the directory if needed.
auto uses /var/log/journal if the directory already exists, and falls back to /run/log/journal if it does not.
none drops logs after forwarding them to other targets, such as a syslog daemon.

On many systemd versions auto is the default, and the whole issue depends on that one condition. With auto, journald does not create /var/log/journal. It only checks whether it exists. Full distribution installs usually create the directory, often through a package script or a tmpfiles rule. Minimal and cloud images are trimmed to stay small and avoid disk writes, and sometimes the directory is missing. Nothing warns you. journald quietly picks the runtime location. Even on systems with persistent logging, journald always starts in /run/log/journal during early boot, because /var may not be mounted yet. Later in boot, systemd-journal-flush.service asks journald to move the runtime journal to /var/log/journal. If the persistent directory is missing, that flush has nowhere to go, so the journal stays in RAM for the whole uptime.

What volatile mode does to memory

/run is a tmpfs, which is a filesystem whose files exist only in memory. When journald appends to a journal file there, the kernel allocates memory pages to hold the data. journald accesses journal files through memory mapping (mmap), so writes go straight into those pages. These pages differ from normal file cache, and that is the part that often confuses people reading free. When a process reads a file on disk, the kernel caches it in RAM and can drop that cache under memory pressure because the data is still on disk. tmpfs pages have no disk copy. They appear as Shmem in /proc/meminfo and under 'shared' and 'buff/cache' in free. The kernel cannot drop them. It can only move them to swap, and many cloud VPS images ship without swap. Without swap, every byte of runtime journal stays in RAM until journald deletes the file. journald does limit itself. RuntimeMaxUse= caps how much space the runtime journal may use, and RuntimeKeepFree= tells it to leave a certain amount of the filesystem free. The defaults are a percentage of the /run filesystem size with an upper cap. When the limit is reached, journald rotates the active file and deletes the oldest archived files. So the journal does not grow without bound, but two problems remain. First, the cap is relative to /run, and /run is itself sized as a fraction of RAM. On a small VPS the default cap is low, so a noisy application can push older entries out within minutes. You get a short log window that you cannot rely on during an incident. Second, the memory used up to the cap is still memory your applications cannot use. On a small instance already near its limit, that can be the difference between running fine and the OOM killer (the kernel's out-of-memory handler) ending a PHP worker or a database process. High log volume also adds work for journald itself. Its rate limiting, controlled by RateLimitIntervalSec= and RateLimitBurst=, drops messages from a service that logs faster than the allowed burst within the interval. That reduces load but can hide exactly the lines you wanted. The reboot problem is simpler. tmpfs is emptied at shutdown, so everything before the current boot is gone. If the VPS rebooted because of a kernel panic, an OOM storm, or a hypervisor event, the logs that would explain it no longer exist.

Confirming the journal is in RAM

A support engineer usually checks three things: whether the persistent directory exists, how many boots the journal records, and how much of /run the journal uses. These commands only read state and are safe on production.

BASH

# Which journal directories exist? Volatile mode usually has only /run/log/journal
ls -ld /var/log/journal /run/log/journal

# Number of boots recorded; one entry after several reboots points to volatile mode
journalctl --list-boots

# Total journal size (runtime and persistent combined)
journalctl --disk-usage

# Size of the /run tmpfs and memory held by tmpfs/shared memory
df -h /run
grep -E '^Shmem:' /proc/meminfo

# Effective journald configuration, including drop-ins under journald.conf.d
systemd-analyze cat-config systemd/journald.conf

# Check whether the OOM killer has been active
sudo dmesg -T | grep -iE 'oom|out of memory|killed process'

If ls reports that /var/log/journal does not exist, --list-boots shows a single boot, and the config output has no explicit Storage= line (or has Storage=volatile), the journal is in RAM. journald also logs its own sizing at startup, usually a line mentioning the runtime journal path, its current size, and its maximum. Running journalctl -b -u systemd-journald typically shows it, although the exact wording depends on the systemd version. When checking memory, compare Shmem with the journal usage. If Shmem is much larger than the journal, something else is using tmpfs or shared memory, and journald is not the main cause.

Turning on persistent storage and capping the runtime journal

The cleanest fix is a drop-in file instead of editing journald.conf directly. Package upgrades can replace the main file, while drop-ins under /etc/systemd/journald.conf.d/ are left alone. Setting Storage=persistent explicitly also removes the dependency on whether the directory happens to exist, because journald creates it. Create /etc/systemd/journald.conf.d/10-storage.conf with your editor:

INI

# Illustrative values: adjust to your disk size and RAM
[Journal]
Storage=persistent
# Upper limit for logs on disk under /var/log/journal
SystemMaxUse=500M
# Upper limit for logs kept in /run/log/journal (early boot, or if /var is unavailable)
RuntimeMaxUse=50M

SystemMaxUse= matters as soon as logs move to disk, because a small VPS root partition can fill just as easily as RAM. If you have dealt with that before, the trade-offs between one-off vacuuming and a permanent cap are covered in systemd Journal Filling /var on a WHM Server: Vacuum Once or Set SystemMaxUse?. RuntimeMaxUse= still applies after you switch to persistent mode. It limits the early-boot journal and any fallback to /run, which is useful on instances where every megabyte of RAM counts. Apply the change and move the current runtime logs to disk:

BASH

# Restart journald so it reads the new drop-in
sudo systemctl restart systemd-journald

# Ask journald to move the runtime journal from /run to /var/log/journal
sudo journalctl --flush

# Verify: a machine-id directory should now exist on disk
ls -l /var/log/journal/
journalctl --disk-usage

Restarting journald is generally safe on current systemd releases, because systemd keeps service log streams open across the restart. On older versions, services sometimes stopped sending stdout logs until they were restarted. If you manage an older host, schedule the restart in a quiet window, or create the directory and wait for the next planned reboot, when systemd-journal-flush.service moves the logs automatically. After the next reboot, journalctl --list-boots should list more than one boot, and journalctl -b -1 should show the previous one. That is the real test that persistence works.

When keeping logs in RAM is a reasonable choice

Volatile mode is not wrong in every case. Some images use it on purpose. Short-lived build workers, autoscaled nodes that ship logs to a central collector, or instances on storage where write volume matters may not need a local history. In those setups, the right fix is to keep volatile mode deliberately: set Storage=volatile explicitly so the next engineer knows it was a decision, set a low RuntimeMaxUse=, and make sure log forwarding actually works. If the collector is down and the node reboots, those logs are lost. The problem is accidental volatile mode, where nobody chose it and nobody notices until a postmortem has no logs to read. When a customer moves a site to a fresh minimal VPS and later asks why the server rebooted, checking journalctl --list-boots early saves time. If it shows only one boot, there is nothing to investigate locally, and the conversation moves to provider-side logs or monitoring. For more ways to reduce journald's footprint on both RAM and disk, see Practical Notes on Minimizing ram and disk used by systemd-journald. The practical habit: whenever you build or adopt a new image, check ls -ld /var/log/journal and journalctl --list-boots before putting real workloads on it. Then decide on purpose. Either set Storage=persistent with a SystemMaxUse= that fits the disk, or keep volatile mode with a tight RuntimeMaxUse= and working log shipping. Both are valid. Leaving it to whether a directory happens to exist is not.