Thinking Certbot Auto-Renews Forever, Until Exhausted Inodes Make It Fail

A common belief after setting up Let's Encrypt with Certbot is that certificates are now handled for good. The package installed a timer, the first renewal went through, and from then on the certificate should refresh itself with nobody watching. That holds until the server runs out of disk space, or more confusingly, out of inodes. From that point every renewal attempt fails, nothing visibly breaks for weeks, and the first sign is a browser warning that the certificate has expired.

What the Certbot timer actually does

On most distro packages, Certbot ships a systemd timer, usually certbot.timer, that starts certbot.service on a schedule, typically twice a day. Snap installs use snap.certbot.renew.timer instead, and some older setups rely on a cron entry. The exact names depend on how Certbot was installed. Each run executes Certbot's renew command. It reads every certificate config under /etc/letsencrypt/renewal/ and renews only the certificates close to expiry, by default within 30 days. Most runs do nothing at all. The timer is a scheduler, not a supervisor. It starts the job and records the exit status. It does not retry with a different strategy, it does not page anyone, and it does not care that the same failure has repeated for three weeks. Automation that never reports failure is only half automated.

Why renewal needs free blocks and free inodes

A renewal is not just a network call to the certificate authority. It writes several files locally. A new private key and certificate land in /etc/letsencrypt/archive/<domain>/. The symlinks in /etc/letsencrypt/live/<domain>/ get updated. The renewal config may be rewritten. A fresh entry goes into /var/log/letsencrypt/letsencrypt.log. With the webroot plugin, a challenge file is created under .well-known/acme-challenge/ in the site's document root. Python may also create temporary files along the way. Each of those files needs two things: data blocks for its content, and an inode. An inode is the filesystem's metadata record for a file: owner, permissions, timestamps, and where the data lives. On ext4 the number of inodes is fixed when the filesystem is created. Once they are used up, no new file can be created, even with gigabytes of free space. XFS and Btrfs allocate inodes more flexibly, so this is mostly an ext4 story, but ext4 is the default on many VPS images. Both conditions produce the same kernel error, ENOSPC, which shows up as No space left on device. This is where people get misled. Someone runs df -h, sees the disk at 60 percent, and decides the error must be wrong. It isn't. The disk ran out of a different resource. The two usual causes look different. Runaway logs fill blocks: an application left in debug mode, a plugin writing the same warning on every request, or an access log that rotation never touches. Inode exhaustion comes from huge numbers of small files: PHP session files that are never garbage collected, cache plugins writing a file per page variant, a stuck mail queue, or years of rotated logs that were never deleted. Certbot also keeps its own rotated logs, by default up to a large number of backups. It is rarely the main cause, but it is one more directory of small files.

Why the failure stays quiet

The 30-day renewal window is meant to be a safety margin. With a twice-daily timer, that is dozens of renewal attempts before expiry. It is also dozens of failures that nobody reads. systemd writes each failure to the journal and marks the service as failed. Cron setups may email the output to root, which on many servers goes to a local mailbox nobody checks. Meanwhile the web server keeps serving the old certificate, which is still valid, so the site looks healthy. Uptime monitors that only check for an HTTP 200 stay green the whole time. Let's Encrypt has also ended its expiration reminder emails, so an inbox warning is no longer a backstop either. From the support side, an expired-certificate report usually sends people straight to DNS, firewall rules, or port 80. Those are reasonable first checks, and the port 80 case is covered in Closed Port 80 for Security, Now Let's Encrypt Renewal Fails. Disk and inode usage is the quick check that often gets skipped, even though it takes seconds.

Finding the error in Certbot's logs

Start with what the timer and service recorded, then search Certbot's own log for the ENOSPC message. All of these are read-only, except the dry run, which talks to the Let's Encrypt staging environment and does not replace your real certificate.

BASH

# When did the timer last fire, and when will it fire next?
systemctl list-timers | grep -i certbot

# Did the last run succeed? (snap installs: snap.certbot.renew.service)
systemctl status certbot.service
journalctl -u certbot.service --since '14 days ago'

# Search Certbot's own log and its rotated copies for the disk error
sudo grep -i 'no space left' /var/log/letsencrypt/letsencrypt.log*

# Show every certificate with its expiry date
sudo certbot certificates

# Test renewal against the staging environment once space is freed
sudo certbot renew --dry-run

In the journal or log, the failure usually ends with a Python traceback. The last line is the useful part.

TEXT

# Illustrative output format, not from a real system
OSError: [Errno 28] No space left on device

If the disk is completely full, Certbot may not be able to write its log at all. In that case the journal is the more reliable record, provided journald itself still has room. If certbot certificates shows an expiry date that is weeks old relative to its last successful renewal, renewals have been failing for a while regardless of what the logs say.

Checking inodes and log rotation

Check blocks and inodes side by side. The column that matters for inodes is IUse%. Anything close to 100 percent on the filesystem holding /etc, /var, or the document root will break renewal.

BASH

# Block usage and inode usage for the root filesystem
df -h /
df -i /

# Which directories hold the most files? Stays on one filesystem. Can be slow.
sudo du --inodes -x -d 3 / 2>/dev/null | sort -n | tail -20

# Which directories hold the most data under /var?
sudo du -xh -d 2 /var 2>/dev/null | sort -h | tail -20

# Dry-run logrotate: shows what would be rotated without changing anything
sudo logrotate -d /etc/logrotate.conf

The logrotate debug output shows which logs have rules and which do not. A log file that grows without limit usually has no matching file in /etc/logrotate.d/, or it has a rule pointing at a path that changed after an upgrade or migration. Be careful with cleanup. Deleting a log that a running process still holds open does not free its space until that process closes the file or restarts. Truncating the file is often safer than removing it. For inode problems, deleting stale session or cache files fixes the immediate issue, but the files come back unless you fix what creates them. Never delete anything under /etc/letsencrypt/ to free space. The archive directory is small, and the live symlinks point into it. Once there is room again, run the dry run, then a real renewal, and confirm the web server picked up the new certificate. Certbot's --deploy-hook option is the usual way to reload Nginx or Apache automatically after a successful renewal.

Making it part of routine maintenance

The fix is not a cleverer timer. It is a small, regular check that someone actually reads. Whenever you review a server, weekly or as part of patching, look at four things: df -h, df -i, the last few runs in the Certbot service journal, and the expiry dates from certbot certificates. That takes under a minute and catches the problem with weeks of margin left. If you have monitoring, alert on inode usage the same way you alert on disk usage. A threshold such as 85 percent works as an illustrative example. Alert on certificate expiry measured from outside the server, by checking the certificate's actual expiry date rather than only whether the page loads. An external check catches every cause of failed renewal, including the ones that have nothing to do with disk. Certbot's timer does its job: it keeps trying. It just has no way to tell you that every try is failing. Treat free blocks, free inodes, and working log rotation as prerequisites for renewal, and check them on a schedule you control.