Why MySQL Crashes When Binary Logs Silently Fill Your Disk

A database server stops accepting writes without warning, your web application begins throwing generic database connection errors, and running df -h in the terminal reveals that the root or database partition is at 100% capacity. When you check the status of the service, mysqld is inactive or stuck in a crash loop. A quick inspection of /var/lib/mysql often shows dozens or hundreds of files named binlog.000001, binlog.000002, or mysql-bin.000001, each taking up hundreds of megabytes or a full gigabyte of space. This pattern is one of the most common storage emergencies on production virtual machines. The database did not suffer a sudden traffic surge or write an enormous amount of application data into its tables. Instead, the server ran out of disk space because MySQL faithfully recorded every single insert, update, and delete into binary log files without anyone instructing it when to discard them.

The Ledger That Keeps Growing

To understand why these files appear, picture a busy accounting office where a clerk writes all financial transactions into a main ledger. In addition to updating the ledger balances, the clerk also writes every single event onto a continuous paper scroll so that if the building catches fire, auditors can recreate every entry from the beginning of the day. The ledger balances stay relatively compact because entries overwrite or sum up previous numbers, but the paper scroll keeps unrolling endlessly. If nobody cuts and shreds old scrolls from last month, the scrolls eventually fill the room until the clerk can no longer sit down. In MySQL, your actual tables and indexes live inside InnoDB data files like ibdata1 or individual .ibd files. When you update a user's status or run a batch job that changes timestamps, the data file updates those specific pages in place. The binary log (often shortened to binlog) is that separate, append-only scroll. It records description events or row changes for every transaction that modifies data, along with execution time and metadata. Engineers often confuse binary logs with InnoDB redo logs. While redo logs help the storage engine recover from unexpected power loss by rolling back unfinished work on restart—a process explored in depth during investigations into Anatomy of an InnoDB Write Stall: Checkpoint Age Thresholds and Synchronous Flush Storms—the binary log serves two different purposes: replication and point-in-time recovery. Read replicas read these binary log events to mirror changes made on the primary server. Similarly, if your primary server suffers hardware failure and you restore a full database backup taken at midnight, you replay binary logs from midnight to the time of failure to recover transactions without losing a day's worth of records. Because binary logs must allow recovery across hours, days, or weeks, MySQL does not overwrite them cyclically like redo logs. When an active binary log file reaches its size limit—typically 1 GB by default via the max_binlog_size variable—MySQL closes it, increments the sequence number, and opens a new file. If no retention policy is configured, MySQL keeps writing new files forever.

Why MySQL Refuses to Keep Running

When a physical disk or cloud block storage volume reaches 100% capacity, standard operating system system calls like write() and pwrite() return an ENOSPC error (No space left on device). Applications handle this error in different ways, but a transactional database engine cannot shrug it off. MySQL guarantees durability through a two-phase commit protocol across its storage components. When a client executes a COMMIT statement, MySQL must guarantee that both the InnoDB storage engine and the binary log agree on the transaction state. First, InnoDB prepares the transaction by writing to its redo log. Next, the MySQL server writes the transaction data to the active binary log on disk and flushes it using fsync(). Finally, InnoDB marks the transaction as committed. If the server attempts to flush the binary log and the underlying filesystem responds with ENOSPC, the atomicity of the transaction is broken. MySQL cannot commit the transaction because it cannot ensure that replicas or point-in-time restores will see the exact same change. Continuing to accept queries would cause silent data drift between the primary node and its replicas. Faced with an unwriteable binary log during a commit, MySQL protects data consistency by halting write operations or deliberately triggering a panic shutdown. Once shut down, starting the service again usually fails immediately. During its initialization routine, MySQL attempts to open temporary files, allocate buffer space, and write startup events to the binary log. When the operating system rejects these file writes due to zero available bytes, the initialization aborts, leaving administrators locked out of the database interface.

The Destructive Instinct of Running rm

A system administrator or support engineer staring at a full filesystem often looks for the largest files and reaches for rm. Running rm -f /var/lib/mysql/binlog.000* from the Linux shell immediately frees several gigabytes of disk space, allowing the server to boot up, but it introduces subtle corruption into MySQL's internal tracking. Alongside the numbered binary log files sits a companion file named binlog.index (or mysql-bin.index). This plain text file contains an ordered list of all binary log segments that MySQL believes currently exist on disk. When you delete binlog files directly through the operating system shell, the index file is not updated. When MySQL boots or attempts to rotate logs later, it consults this index file. If it encounters listed files that no longer exist, commands like SHOW BINARY LOGS can return errors, automated backup utilities that stream logs can fail midway, and connected replicas will stall if they request an event from an index position whose underlying file was wiped. Furthermore, if you accidentally delete the active log file that was receiving writes at the moment of the crash, you eliminate the only record needed to reconcile incomplete transactions during InnoDB crash recovery. Files managed by the database daemon should always be inspected and purged through database commands, not through raw filesystem deletions.

Safely Freeing Space and Automating Expiration

If your database is completely stopped and refuses to start because there are zero free bytes on the filesystem, you must free enough headroom for the process to complete startup tasks. Instead of deleting binary logs with rm, look for uncompressed rotated operating system logs under /var/log, clear package manager caches (apt-get clean or yum clean all), or temporarily move non-database backup archives to another volume. If you must delete something inside the MySQL directory as a last resort, removing old error log files or discarded .sql dumps left behind by manual imports is much safer than touching binary log files. Once MySQL is running, clean up older binary log files safely using the built-in PURGE BINARY LOGS statement. You can specify an explicit log sequence number or a date threshold. Afterward, set up automatic retention so the issue does not recur.

SQL

-- Step 1: Check existing logs and their sequence numbers
SHOW BINARY LOGS;

-- Step 2: Purge older logs safely (keeps index file consistent)
PURGE BINARY LOGS TO 'binlog.000042';
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;

-- Step 3: Inspect current automatic retention settings
SHOW VARIABLES LIKE 'binlog_expire_logs_seconds';

-- Step 4: Configure retention to 3 days (259200 seconds) at runtime
SET GLOBAL binlog_expire_logs_seconds = 259200;

The PURGE BINARY LOGS TO command deletes all log files listed in the index file prior to the designated filename, updating the index safely without breaking references. The BEFORE variation performs the same cleanup based on a timestamp relative to the present moment. To make this retention permanent across reboots, define the expiration variable inside your MySQL configuration file (typically /etc/mysql/my.cnf, /etc/my.cnf, or a configuration drop-in file under /etc/mysql/mysql.conf.d/): In MySQL 8.0, configure retention using seconds:

INI

[mysqld]
binlog_expire_logs_seconds = 259200
max_binlog_size = 1073741824

If you are operating legacy MySQL 5.7 instances, the variable is measured in days instead:

INI

[mysqld]
expire_logs_days = 3
max_binlog_size = 1073741824

When binlog_expire_logs_seconds (or expire_logs_days) is set to a non-zero value, MySQL automatically evaluates and purges expired binary logs whenever the active log rotates or when the server starts up. Setting this value to 0 disables automatic purging entirely, which is the exact configuration trap that leads to unmonitored disk exhaustion. When selecting a retention window, match your timeframe to your operational needs. If your backup strategy takes full nightly dumps and pulls binary logs every hour for point-in-time recovery, keeping three days of binary logs locally provides ample buffer while keeping storage usage predictable. If you run asynchronous read replicas, make sure your retention window is wide enough that a replica lagging behind by twenty-four hours due to maintenance can catch up without finding its required log position purged from the primary node. Binary logs are critical tools for data safety and high availability, but treating them as permanent archives on your primary storage volume turns a reliability mechanism into a point of failure. Setting an explicit expiration window and relying on native purge commands prevents disk capacity emergencies while keeping your point-in-time recovery path intact.