When Keepalives Vanish: Debugging IPsec MTU Black Holes Across Cloud Transit Gateways

A network link can simultaneously boast zero packet loss on your monitoring dashboard and fail to pass a single byte of application traffic. Every network engineer eventually runs into this ghost in production: an IPsec or GRE tunnel connecting an on-premises core router to an AWS Transit Gateway (TGW) or virtual cloud appliance reports pristine latency and full ICMP reachability, yet the Border Gateway Protocol (BGP) session systematically cycles from Established to Active and drops into Idle. Synthetic ping probes report 100% uptime, but every couple of minutes, the BGP hold timer drains to zero, tearing down routing tables across the hybrid infrastructure. This exact failure mode was dissected in the operational teardown BGP Route Flapping: Diagnosing Multi Cloud Transit Gateway Partitions, exposing how multi-cloud transit architectures routinely collapse under silent Path MTU (PMTU) black holes. Teams often mistake these flaps for control-plane instability, memory pressure on border routers, or provider physical cutovers. In reality, the breakdown lives squarely at Layer 3 and Layer 4 encapsulation boundaries where standard packet-sizing assumptions silently shatter.

The Anatomy of an Encapsulated Black Hole

Under normal idle conditions, BGP keepalive packets are tiny—just 19 bytes of BGP payload nested inside a 20-byte TCP header and a 20-byte IPv4 header. Even after wrapping that 59-byte packet in Generic Routing Encapsulation (GRE) or an IPsec Encapsulating Security Payload (ESP) tunnel with initialization vectors and HMAC trailers, the entire wire frame rarely exceeds 150 bytes. It slips effortlessly through any interface, regardless of how aggressively intermediate links constrain the maximum transmission unit.

TEXT

Standard Frame (1500 bytes) + ESP/GRE Overhead (50-76 bytes) = 1550+ bytes
[ Outer IP (20B) | ESP Header (8B) | IV (8-16B) | Inner IP (20B) | TCP (20B) | Payload | ESP Trail/Auth (16B) ]
                                                                          +--- Exceeds Cloud 1500 MTU ---> [DROP]

The catastrophe strikes during state changes. When a routing event triggers an influx of Network Layer Reachability Information (NLRI), or when workload hosts initiate high-throughput data streams, TCP naturally expands its segment size to utilize the interface Maximum Segment Size (MSS). If the local Ethernet interface has a standard MTU of 1500 bytes, TCP will negotiate an MSS of 1460 bytes. Once the tunnel adds 50 to 76 bytes of IPsec ESP tunnel-mode overhead (or GRE plus IPsec transport mode), the resulting transit packet demands 1550 to 1576 bytes of physical wire space. When this oversized frame hits a cloud transit gateway or virtual firewall enforcing a strict 1500-byte or 1420-byte interface limit, the transit node must either fragment the packet or discard it. Because nearly all production IP stacks set the Don't Fragment (DF) bit on TCP segments to prevent the CPU serialization penalties of reassembly, the transit gateway must drop the frame and originate an ICMP Type 3, Code 4 message: Destination Unreachable, Fragmentation Needed and DF Set.

Why Path MTU Discovery Fails in the Cloud

In an ideal RFC 1191 world, the originating endpoint receives the ICMP Type 3 Code 4 packet, extracts the proposed Next-Hop MTU from the ICMP payload, caches the smaller MTU value in its kernel routing table, and retransmits the segment with a lower MSS. The session adapts dynamically without operational intervention. In the real world of hybrid cloud networks, ICMP is treated as a security pariah. Network administrators and default cloud security group configurations blanket-filter inbound ICMP, assuming all ICMP packets are reconnaissance probes or denial-of-service vectors. Worse, several proprietary virtual appliances and stateless load balancers silently discard transit frames exceeding their Maximum Transmission Unit without generating an ICMP unreachable reply at all. When the transit gateway drops the oversized BGP Update packet and drops or filters the corresponding ICMP Type 3 Code 4 notification, the sender has no mechanism to learn that the frame vanished. The sender retransmits the identical 1460-byte payload at exponential backoff intervals. Meanwhile, the remote BGP peer receives nothing. It maintains the peering as long as small keepalives trickle through, but the moment the hold timer expires—often negotiated down to 30 seconds to match cloud defaults—the remote peer assumes the connection is dead, issues an internal hold timer expiration, and tears down the session.

Diagnostic Verification via Packet Capture

Diagnosing a PMTU black hole requires looking beyond ICMP echo tests. If ping -c 5 <remote_ip> works, rerun the probe while forcing the Don't Fragment bit and stepping up the payload size. On Linux, this is done using the -M do flag:

BASH

# Probe tunnel path without fragmentation using explicit payload sizes
ping -M do -s 1472 10.254.0.1  # 1472 + 28 (IP/ICMP) = 1500 bytes
ping -M do -s 1392 10.254.0.1  # 1392 + 28 (IP/ICMP) = 1420 bytes

If the 1392-byte ping succeeds but the 1472-byte probe yields Frag needed and DF set locally or hangs in total silence without packet return, you have isolated an MTU barrier. Running tcpdump on the tunnel endpoint will reveal the BGP TCP handshake completing successfully, followed by an abrupt sequence of unacknowledged retransmissions matching the maximum segment size immediately preceding the peer timeout.

Enforcing Deterministic TCP MSS Clamping

Relying on Path MTU Discovery across disparate cloud domains and stateful inspection tiers is an anti-pattern. The deterministic engineering fix is enforcing TCP MSS clamping directly on all tunnel-facing virtual interfaces, forcing both ends of a TCP handshake to negotiate down to an MSS that cleanly accommodates all outer encapsulation layers. For a standard IPsec ESP tunnel (with AES-GCM-256 and SHA-256) running over a base 1500-byte underlay, an inner interface MTU of 1400 bytes provides adequate headroom for tunnel headers, NAT-Traversal (UDP 4500) encapsulation, and dynamic routing wrappers. Clamping the MSS to 1360 bytes (1400 bytes minus 40 bytes for IPv4 and TCP headers) completely eliminates payload-driven fragmentation. On modern Linux-based edge routers or firewalls running nftables, apply MSS clamping explicitly on the forward chain for all tunnel interfaces:

BASH

# Clamping MSS to 1360 on outgoing traffic traversing the IPsec VTI
nft add table ip filter
nft add chain ip filter forward { type filter hook forward priority 0 \; }
nft add rule ip filter forward oifname "vti0" tcp flags syn tcp option maxseg size set 1360

If using legacy iptables, the equivalent rule intercepts TCP SYN packets and clamps the MSS to match the path automatically:

BASH

# Automatically clamp MSS to interface MTU minus IP/TCP overhead
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -o vti0 -j TCPMSS --clamp-mss-to-pmtu

Beyond TCP clamping, the underlying tunnel interface itself (such as a Virtual Tunnel Interface vti0 or GRE tunnel gre1) must have its base MTU configured manually to match that inner limit (e.g., ip link set dev vti0 mtu 1400). This ensures non-TCP control-plane protocols, including OSPF or non-TCP BFD probes, do not attempt to transmit frames that exceed transit boundaries.

Structural Hardening for Cloud Gateways

Tunnel MTU sizing cannot be solved in isolation from cloud-side path design. When routing through AWS Transit Gateway with Third-Party Firewall Appliances, Availability Zone asymmetry frequently compounds MTU issues. Because AWS TGW maintains AZ affinity by default, return traffic can land on a firewall instance in AZ-B that never saw the initial SYN packet traversing AZ-A, causing stateful firewalls to discard out-of-order or seemingly unestablished TCP segments. To harden hybrid transit backbones:

Explicitly allow ICMP Type 3 Code 4 in all perimeter security groups and cloud network access lists (NACLs) to allow PMTUD to function whenever clamping rules fail to match abnormal payloads.
Enable Appliance Mode on Transit Gateway VPC attachments when routing east-west or north-south traffic through intermediate network inspection VPCs, ensuring bidirectional symmetry across AZ boundaries.
De-couple control plane BFD from underlay links. Avoid running aggressive sub-second Bidirectional Forwarding Detection timers on the unencapsulated Direct Connect underlay if the overlay IPsec tunnels are relying on independent, loose BGP hold timers. A mismatch here creates a 30-second routing black hole during failovers while the overlay waits to expire.

Treating network health as a binary alive-or-dead metric based on ICMP ping reachability is an operational trap. In an era of layered multi-cloud overlays, packet size is an active variable in state survival. Clamping MSS and explicitly designing for the worst-case encapsulation overhead guarantees that when control-plane updates surge, your BGP keepalives won't vanish into the void.