Why New MySQL Connections Stall for Seconds Before Login on a Remote DB Server

A site moves its database to a separate server, and pages start loading a few seconds slower. Slow query logs are empty and EXPLAIN looks fine. Running the same queries from a console that is already connected returns instantly. The time is being lost before the query ever runs, inside the step where the client opens a new connection and logs in. When that delay is a near-constant few seconds per connection, the usual cause is DNS. Specifically, it is the MySQL server trying to turn the client's IP address into a hostname. This article walks through what the server does during connection setup, why a resolver problem on the database host turns into page latency on the web host, and what to check before switching name resolution off.

What the server does before it says hello

A MySQL connection over TCP goes through a few stages. The client sends a SYN and the kernel on the DB server completes the three-way handshake. At that point the socket is in the listen queue, mysqld calls accept(), and the connection is handed to a thread. The next part is where people expect the server to start talking right away. In the MySQL protocol, the server speaks first. It sends a greeting packet with its version, a salt (random bytes used for the password challenge), and the authentication plugin. The client cannot send a username until it receives that greeting. Before building the greeting, the connection thread has to decide which accounts could even apply to this client. Accounts in MySQL are user@host pairs, and the host part can be an IP, a subnet pattern, a wildcard like %, or a hostname like web01.internal. To match hostname-based accounts, the server needs a hostname. So, unless name resolution is turned off, it does roughly this:

Reads the peer address of the socket, for example 10.0.0.10 (illustrative example).
Performs a reverse lookup (a PTR query, which asks DNS which name belongs to an IP) through the system resolver, using getnameinfo().
If a name comes back, performs a forward lookup of that name with getaddrinfo() and checks that one of the returned addresses is the original IP. This is forward-confirmed reverse DNS. It stops someone who controls their own PTR records from claiming to be trusted-host.example.
Stores the result in the host cache, checks the host against the grant tables, and only then sends the greeting.

These are blocking calls in the connection thread. If the resolver answers in a millisecond, nobody notices. If it waits on a nameserver that never replies, the thread sits there and the client sits there with it. From the client side, the TCP connection is established but no bytes arrive.

Why the delay is so regular

The system resolver on Linux is glibc's stub resolver, configured through /etc/resolv.conf and /etc/nsswitch.conf. The hosts: line in nsswitch usually says files dns, so /etc/hosts is checked first and DNS second. When a query goes to a nameserver that drops it, glibc waits for its timeout and then retries. The documented defaults are a 5 second timeout and 2 attempts, spread across the listed nameservers. That is why this problem tends to show up as stalls in clean multiples of about five seconds rather than random jitter. If the first nameserver in the list is dead and the second works, every lookup pays the first timeout and then succeeds. That case is easy to miss, because DNS technically works. The usual triggers on a split web/DB setup are:

The web server talks to the DB over a private network, and nobody created PTR records for those private addresses. The query gets forwarded to a resolver that has no zone for them and either times out or answers slowly.
The DB server's resolv.conf points at a resolver that is unreachable from that host, often because of a firewall rule, a VPN change, or a migration that copied config from another network.
A local caching resolver on the DB host has died, and the fallback entry is far away.

If the stall gets longer than connect_timeout (10 seconds by default on MySQL), the server gives up on the handshake. The client then sees Lost connection to MySQL server at 'reading initial communication packet', which is a strong hint that the wait happened before the greeting. A similar pattern on the web side, where workers stall on outbound lookups, is covered in Why Using Public Resolvers in resolv.conf Stalls Apache Workers. The host cache should, in theory, make this a one-time cost per client IP. In practice, it often does not. A lookup that times out is usually classed as a transient error, and depending on the version the server may try again on later connections instead of trusting the failure. The cache can also be flushed or sized down to zero. Meanwhile, a typical PHP site opens a fresh connection on every request, so any per-connection cost lands on every page view.

Confirming it from the database host

Test from the DB server, not the web server. The web server's resolver is irrelevant here, because the lookup that matters runs inside mysqld. Replace the IP with your web server's private address.

BASH

# Run on the database server. 10.0.0.10 is an illustrative web server IP.
# getent uses the same nsswitch/resolver path as mysqld
time getent hosts 10.0.0.10

# Ask DNS directly, with short limits so it fails fast
dig -x 10.0.0.10 +tries=1 +time=2

# Which resolvers and options this host is actually using
grep -E '^(nameserver|options|search)' /etc/resolv.conf

If getent takes several seconds, or dig times out against the first nameserver, you have found the delay. A fast NXDOMAIN (no such name) is not the problem. A fast negative answer costs almost nothing. What hurts is no answer. You can also see this from inside MySQL. While a slow connection is in progress, SHOW PROCESSLIST typically lists it as unauthenticated user with the client IP as the host, before any username appears. On versions with Performance Schema enabled, the host cache records what happened to each lookup:

SQL

-- Per-client-IP record of name resolution results
SELECT IP, HOST, HOST_VALIDATED,
       COUNT_NAMEINFO_TRANSIENT_ERRORS,
       COUNT_NAMEINFO_PERMANENT_ERRORS,
       COUNT_FORWARD_TRANSIENT_ERRORS,
       SUM_CONNECT_ERRORS
FROM performance_schema.host_cache;

Transient error counters that keep climbing for your web server's IP match a resolver that is timing out. HOST being NULL with permanent errors means the lookup failed cleanly, which is usually harmless for speed. Column names have changed across releases and forks, so check the table definition on your version if the query errors. In support, a useful tell is the shape of the complaint. Problems with slow queries vary with the page. This one adds roughly the same few seconds to every uncached page, including trivial ones, and it often starts right after a database migration or a network change rather than after a code deploy.

Fixing the lookup versus skipping it

There are two kinds of fix. You can make the lookup fast, or you can tell MySQL not to do it. Making it fast is the lower-risk change. Correct the DB server's resolv.conf so the first nameserver is reachable. Alternatively, add the web servers to /etc/hosts on the DB server, with lines like 10.0.0.10 web01.internal. Because nsswitch checks files first, both the reverse and forward lookups are answered locally, and the forward-confirm step passes. Proper PTR records for the private range work too, if you control that DNS. None of these touch MySQL accounts. The other option is skip_name_resolve, set under [mysqld] in the server config. With it on, the server never calls the resolver during login and matches clients by IP only. On most versions this setting is not dynamic, so it needs a restart. For a dedicated DB server reached by a known set of app servers, it is a common choice. It removes a dependency on DNS from every login. The catch is the grant tables. Once names are not resolved, any account whose host part is a hostname or a domain pattern can no longer match anyone. Look for those accounts before you restart:

SQL

-- Accounts whose host part contains letters (likely hostnames or name patterns).
-- IPv6 addresses contain hex letters too, so review the results by eye.
SELECT user, host
FROM mysql.user
WHERE host NOT IN ('localhost', '%')
  AND host REGEXP '[g-zG-Z]|[a-fA-F]{2,}[.]';

For each hit, create the equivalent account using the client's IP or subnet, such as 'app'@'10.0.0.10' or 'app'@'10.0.0.%', with the same privileges. Drop the old one only after the app is confirmed to work with the new one. The same hostnames can appear in database-level and table-level privileges and in DEFINER clauses on views, triggers, and stored procedures, so check those too. Some versions log a warning at startup listing accounts that become unusable, which is worth reading after the restart. Two more details catch people out. First, localhost still matches socket connections with name resolution off, but an app that connects over TCP to 127.0.0.1 matches accounts for 127.0.0.1, not localhost. On a remote DB this rarely matters for the app, but it can break local admin tools or backup scripts. Second, skipping resolution does not fix the resolver. Other things on that host, such as package updates, mail, and monitoring agents, still use it, so the broken nameserver entry is worth correcting anyway. When connections stall for a steady few seconds and queries are fast, run getent hosts against the client IP on the database server first. If that is slow, fix the resolver or add /etc/hosts entries for the app servers. Turn on skip_name_resolve only after every hostname-based grant has an IP-based replacement.