Raised ulimit but the Service Still Hits Too many open files? Check LimitNOFILE

You raised nofile in /etc/security/limits.conf, logged out and back in, and ulimit -n now shows a large number. Yet the web server or proxy still writes Too many open files to its log under load, and new connections still fail. The edit did take effect, but only for your shell. The service runs with a different limit, set somewhere else. This catches people often because the advice to edit limits.conf is everywhere and it used to be the right answer on older init systems. On a systemd-based distro it usually is not, at least for daemons.

What the error is actually telling you

On Linux, almost everything a process holds open is a file descriptor. That includes regular files, log files, pipes, and every network socket. A busy network service holds one descriptor per client connection, one per upstream or backend connection, plus its own logs and listening sockets. It adds up fast. Too many open files is the text for the error code EMFILE (errno 24). It means this one process has reached its own per-process limit, called RLIMIT_NOFILE. In Nginx it typically shows up as accept4() failed (24: Too many open files). The worker could not accept a new connection because it had no free descriptor slot. There is a similar message, Too many open files in system, which is ENFILE. That one is about the kernel-wide pool controlled by fs.file-max. People often go straight to raising fs.file-max with sysctl when they see either message. If your log says Too many open files without in system, the system-wide pool is not the problem. Raising it changes nothing for this case, so you can set it aside. Every process has two values for this limit. The soft limit is what is enforced right now. The hard limit is the ceiling the soft limit can be raised to without extra privileges. When the error appears, the soft limit is the one that was hit.

Why limits.conf never reached the service

/etc/security/limits.conf and the files in /etc/security/limits.d/ are read by pam_limits, a PAM module. PAM runs during login-style events: SSH sessions, console logins, su, sudo, and cron jobs on many setups. When you log in, PAM applies those limits to your session, and everything you start from that shell inherits them. That is why ulimit -n looks correct. A systemd service is not started from a login session. PID 1 forks and execs it directly, with limits taken from the unit file or from systemd's own defaults. PAM is not involved, so limits.conf is never read. On many current distributions the default soft limit for services is still 1024. The hard limit is often much higher, but the soft value is the one that matters at runtime. There is a second trap. If you test by starting the binary by hand from your shell, it inherits your raised shell limit and works fine. Then you run systemctl start and the problem comes back. Both observations are correct. They are just different processes with different limits. In support work this is one of the first things to rule out when someone says they already raised the limit. The question is not whether they edited the file. The question is what the running process actually has.

Read the limit the running process really has

Skip ulimit here, since it only describes your current shell. The kernel exposes each process's live limits under /proc, and that is the value that counts. Find the main PID, read its limits, and compare with how many descriptors it holds right now.

BASH

# Main PID of the service (replace nginx with your unit name)
systemctl show -p MainPID --value nginx

# Live soft and hard limits of that process (replace 1234 with the PID)
grep -i 'open files' /proc/1234/limits

# Number of descriptors the process holds right now
sudo ls /proc/1234/fd | wc -l

# What systemd has configured for the unit
systemctl show -p LimitNOFILE nginx

The /proc line looks like this. The columns are soft limit, hard limit, and units.

TEXT

# Illustrative output format
Max open files            1024                 524288               files

If the soft value is 1024 and the descriptor count sits close to it during busy periods, you have found the cap that is being hit. For multi-process servers, also check the workers and not only the master. Nginx workers, PHP-FPM pool children, and similar processes can have their own limits. Nginx, for example, can raise the worker limit with its own worker_rlimit_nofile directive. Use pgrep -f or ps to find a worker PID and read its /proc/PID/limits as well.

Raise the limit, or find the leak first

Before raising anything, decide whether the process genuinely needs more descriptors or is failing to release them. A higher limit fixes the first case. In the second case it only delays the error. The descriptor count is the clue. Check it a few times across a normal day. If it rises and falls with traffic and peaks close to the limit only at busy times, the workload has outgrown the default, and raising LimitNOFILE is the right move. If it climbs steadily and never drops back, even when traffic is quiet, something is leaking. To see what the descriptors are, sudo ls -l /proc/1234/fd shows each one. Sockets appear as socket:[inode], and files show their path. If they are mostly sockets, look at connection states with ss. sudo ss -tanp state close-wait lists TCP connections in CLOSE_WAIT. That state means the remote side closed the connection but the local application never called close() on its end. A pile of CLOSE_WAIT sockets owned by your service usually points to an application bug, often a backend client or connection pool that does not clean up after errors or timeouts. Restarting clears it for a while. Raising the limit makes the restarts less frequent. Neither fixes it. A rough rule of thumb: if the count tracks traffic, raise the limit. If it tracks uptime, fix the leak and raise the limit modestly while you do. If production is failing right now and you cannot restart, prlimit --pid 1234 --nofile=65535:65535 from util-linux can change the limit of the running process. Treat that as first aid only. It is lost on the next restart and does not touch the unit configuration.

Setting LimitNOFILE in a drop-in and confirming it

Avoid editing the unit file shipped by the package, usually under /usr/lib/systemd/system/ or /lib/systemd/system/. Package upgrades can overwrite it. Use a drop-in instead: a small override file that systemd merges on top of the original. Running sudo systemctl edit nginx opens an editor and saves the result as /etc/systemd/system/nginx.service.d/override.conf.

INI

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
# Applies soft and hard to the same value; use soft:hard to set them separately
LimitNOFILE=65535

The value 65535 here is an illustrative example, not a recommendation. Size it from the descriptor counts you observed, with headroom. Very large values have small costs, and some software allocates per-descriptor structures based on the limit. systemctl edit reloads systemd's configuration for you. If you wrote the file by hand, run sudo systemctl daemon-reload. Then do a full sudo systemctl restart nginx, not a reload. Limits are applied when a process is executed. A reload keeps the same master process, and new workers inherit its old limit. Confirm in three places, in this order:

systemctl show -p LimitNOFILE nginx should show the new value. If it does not, the drop-in was not picked up. Check the path and the [Service] header, and look for it in systemctl cat nginx.
Get the new main PID and check /proc/PID/limits. The PID changes after restart, so do not reuse the old one.
Check a worker or child PID too, because application-level settings can still override what systemd passes in.

That last step matters more than it sounds. Some daemons cap or reset their own limit. MySQL and MariaDB, for example, have an open_files_limit setting that interacts with the systemd value. If the unit shows the new limit but the process does not, look at the application's own configuration next. The practical habit is simple: when a service reports Too many open files, read /proc/PID/limits for the actual process before editing anything. If the soft limit is the systemd default, set LimitNOFILE in a drop-in, restart fully, and read /proc again. If the descriptor count keeps rising regardless of traffic, spend the effort on finding the socket leak, because no limit is high enough for a process that never closes what it opens.