df -h says the root filesystem is at 100%. You run du across the whole disk and the total comes out far smaller, maybe by tens of gigabytes. You delete a big log file to make room, and df does not move at all. This is one of the more confusing disk problems on Linux, and it nearly always has the same cause: a file that was deleted while a running process still had it open.
This article walks through it the way a post-mortem would: what the impact looks like, how the investigation usually goes, the moment the cause becomes clear, how to get the space back without a reboot, and what to change so it does not happen again.
Initial impact: writes failing with no obvious culprit
No space left on device if they can still log at all. On a hosting server, the first thing customers usually notice is a site that suddenly errors on form submits or a control panel that will not load.
The natural reaction is to find big files and remove them. That is where this incident gets stranger, because the big file you remove often turns out to be the very thing that keeps the disk full.
Investigation: df and du disagree
BASH
# Block usage as the filesystem itself reports it
df -h /
# Sum of everything reachable by name, staying on this one filesystem (-x)
sudo du -xsh /
# Inode usage, to rule out inode exhaustion
df -i /
# Open files whose link count is below 1, meaning deleted but still open
sudo lsof -nP +L1df asks the filesystem how many blocks are allocated. du walks directories and adds up the size of every file it can reach by name. Anything that holds blocks but has no name in any directory is counted by df and invisible to du.
A few other things can open a gap between them, and it is worth ruling them out quickly:
- Inode exhaustion. If
df -ishows IUse% at 100%, the disk can report free space and still refuse new files, because every inode (the metadata record each file needs) is taken. Lots of tiny files, such as session files or mail queue entries, cause this. It is a different problem with a different fix. - Files hidden under a mount point. If something was written to a directory before another filesystem got mounted on top of it, those files still use space on the parent filesystem but
ducannot see them. - Reserved blocks. ext4 usually keeps a slice of the disk reserved for root, so
dfcan show the disk as full for normal users before Used reaches Size. This explains a small gap, not a large one.
lsof +L1 line is the one that matters.
The discovery: a file with no name still holding blocks
rm does not directly free its data. It removes a directory entry, which is just a name pointing at an inode. The inode and its data blocks are released only when two things are true: no names point at it anymore (the link count is zero) and no process has it open.
So when a service like Nginx, PHP-FPM, or an application worker has a log file open for writing and someone runs rm on that log, the name disappears but the process keeps its file descriptor. It can keep writing to the file indefinitely. du cannot see it because there is no name to walk to. df still counts every block, and the count keeps growing.
lsof +L1 lists open files with a link count below one, which is exactly this case. The output looks something like this:
TEXT
Illustrative output format:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
php-fpm 1234 www-data 3w REG 253,1 10737418240 0 131075 /var/log/app/debug.log (deleted)
php-fpm 1235 www-data 3w REG 253,1 10737418240 0 131075 /var/log/app/debug.log (deleted)(deleted) after it. SIZE/OFF shows how big the file is, in bytes. FD is the file descriptor number with a mode letter, so 3w means descriptor 3, open for writing. Notice that the same inode can appear under several PIDs, usually because worker processes inherited the descriptor from a parent. The space comes back only once every one of them lets go.
This is the moment the incident makes sense. The disk is not full of mystery data. It is full of a log that was deleted on purpose, by a person or a script, while its writer was still running.
On hosting servers this pattern often comes from a well-meant cleanup: someone sees a huge log, deletes it, and assumes the job is done. If your services run in containers, the same symptom can come from container logs instead, which is covered in Cara Memeriksa Log Container Docker yang Bikin Disk VPS Penuh.
Mitigation: freeing the space without a reboot
BASH
# 1. Confirm the descriptor points at the deleted file (use your PID and FD number, without the mode letter)
sudo ls -l /proc/1234/fd/3
# 2a. Release the blocks by truncating the file through the open descriptor
sudo truncate -s 0 /proc/1234/fd/3
# 2b. Or restart or reload the service holding it so it closes the old file
sudo systemctl restart your-service-name/proc/PID/fd/N is a live handle to the open file, even though it has no name on disk. Truncating it to zero frees the blocks immediately, and the process keeps running without noticing. If the same inode is held by several PIDs, truncating through any one of them is enough, since they all point at the same file.
Before truncating, ask whether you need the contents. Truncation destroys them. If the log might matter for debugging or an audit, copy it out first with something like sudo cp /proc/1234/fd/3 /root/recovered-debug.log, making sure the destination is on a filesystem that actually has room.
There is one side effect to know about. A process that writes without append mode keeps writing at its old offset after the truncate. The file then looks huge in ls because of the gap, but that gap is a sparse region that uses no disk blocks. df tells the truth here; the apparent size does not.
Option 2b is cleaner when you can afford it. Restarting the service makes it close the deleted file and open a fresh one at the normal path. Many daemons also reopen their logs on reload or on a specific signal, which avoids dropping connections, but that behavior depends on the software, so check its documentation rather than assuming. Service names also vary by distro and PHP version, so confirm the unit name with systemctl list-units first.
After either option, run df -h / and sudo lsof -nP +L1 again. The space should be back and the deleted file should be gone from the list.
The lasting fix: rotate logs in a way the writer knows about
rm.
The standard tool for this is logrotate, and it gives you two approaches with a real trade-off between them.
The first is to rename the log and then tell the service to reopen its files, using a postrotate script that runs a reload or sends the signal the daemon expects. This loses no data, because the process finishes writing to the old file and starts cleanly on the new one. It requires the service to support reopening logs, and the script has to be correct for your setup.
The second is copytruncate, which copies the log and then truncates the original in place. The writer never needs to know, which makes it convenient for applications that cannot reopen logs. The cost is a small window between the copy and the truncate where new lines can be lost. For debug or access logs that is usually acceptable. For anything you need to be complete, prefer the reopen approach.
A few other habits close the remaining gaps. Put size limits on rotation, not just a daily schedule, so a runaway debug log cannot fill the disk between runs. Check whether systemd-journald has a size cap, typically set with SystemMaxUse in its config, since defaults vary by distro. Turn off verbose application debug logging once you are done with it, because it is a frequent source of this exact growth. And teach the team that when a live log needs emptying right now, truncating it in place (for example sudo truncate -s 0 /var/log/app/debug.log) frees the space, while rm does not.
Finally, add a check. Disk alerts based on df will catch this, but adding lsof +L1 to your first-response routine is what turns a confusing hour into a short one.
The practical takeaway: when df and du disagree by a lot, check df -i to rule out inodes, then run sudo lsof -nP +L1 before deleting anything else. If a deleted file is still open, free it by truncating through /proc or by restarting the process that holds it, and then fix log rotation so files are reopened or truncated instead of removed out from under a running service.