Deciding If Inter-VPC Timeouts Are Asymmetric Drops or a Dead Path

A support ticket arrives describing a database replica in one VPC that suddenly cannot talk to its primary in another region. The engineer on duty runs a ping test, sees responses returning with low latency, and closes the ticket as an application-level timeout. An hour later, the ticket reopens because API calls between the two environments are still hanging completely. When traffic between two virtual private clouds (VPCs) stalls, our instinct is often to treat the problem as binary: either the network path is up or it is dead. In modern cloud fabrics, network paths are rarely that simple. A dead link is straightforward to diagnose because traffic stops moving in both directions. The harder, more common problem is an asymmetric routing drop, where packets leave on one path, return on another, and get silently discarded along the way by a stateful firewall or a strict routing policy. Understanding how to tell these two failure modes apart saves hours of guessing and prevents pointless finger-pointing between cloud providers and application teams.

Why Inter-VPC Traffic Diverges

To understand why a connection hangs, we have to look at how software-defined networks (SDNs) steer traffic. In a standard bare-metal setup, two routers communicate over physical cables using interior routing protocols, and both forward packets along symmetric paths unless somebody explicitly configures complex policy routing. In cloud platforms, VPC peering, transit gateways, and regional hubs decouple the underlying physical hardware from the routing tables presented in your management console. When you link VPCs across regions or transit hubs, routing relies on encapsulation protocols such as Geneve or VXLAN running over a shared physical spine. Routing tables inside these virtual transit hubs evaluate routes independently in each direction. Subnet A in Region 1 has a route table that points traffic destined for Subnet B toward a transit gateway attachment. Meanwhile, Subnet B in Region 2 might point its return traffic to a different attachment, an equal-cost multi-path (ECMP) group, or a centralized inspection VPC containing virtual firewall appliances. Symmetry breaks when one leg of a multi-path group experiences a partial partition, or when someone updates a route table on one end without mirroring the change on the other. For instance, if an egress route table balances traffic across two transit attachments, but one attachment loses its internal route propagation, outbound packets still travel through Path A. Return packets, however, get routed back through Path B. To the transport layer, both paths exist and have valid routing entries. To the stateful devices sitting along the path, this split is fatal.

Dead Paths vs Asymmetric State Drops

The fundamental difference between a dead path and an asymmetric drop lies in which side discards the packet and why. A dead path means a hop along the route cannot forward the packet to the next hop. This happens when an inter-region transit peering link suffers an internal control-plane failure, a BGP session between virtual routers drops without a fallback, or an underlying tunnel interface goes down. When a path is truly dead, packets leaving the client never reach the destination server. If you send a TCP SYN packet, it vanishes inside the transit hub. No device processes it, no server logs it, and the client simply retransmits until the connection times out. An asymmetric drop looks entirely different at the packet level, even though the end-user symptom—a connection timeout—feels identical. In an asymmetric drop, the client's TCP SYN packet successfully traverses the transit network and arrives at the destination server. The destination server accepts the SYN, creates an embryonic connection state, and replies with a TCP SYN-ACK packet. However, the return path differs from the inbound path. If the return traffic hits a stateful firewall or an intermediate virtual appliance that never saw the initial SYN packet, that appliance drops the SYN-ACK immediately. A stateful firewall tracks connection states: it expects to see SYN, then SYN-ACK, then ACK. If a standalone SYN-ACK arrives out of nowhere on an interface, the firewall treats it as an invalid or malicious packet and drops it silently. The destination server keeps retransmitting the SYN-ACK, while the client keeps retransmitting the initial SYN. Neither side understands why the other is not answering. A similar drop happens directly on Linux hosts when Reverse Path Forwarding (RPF) filtering is active. If the destination server receives a packet on eth0, but the server's own routing table indicates that the best path back to the source IP address is out through eth1, strict RPF considers the packet spoofed and drops it inside the kernel before it even reaches the listening socket. For engineers dealing with MTU issues over transit links, this behavior is often confused with fragmentation problems. You can explore how MTU black holes behave under transit gateways in our notes on debugging IPsec MTU black holes across cloud transit gateways.

Capturing Bidirectional Evidence

You cannot diagnose an asymmetric drop from a single machine. Running traceroute or curl from the source host only shows that the connection fails; it cannot tell you if the request failed to arrive or if the response failed to return. You must collect packet captures concurrently on both the source and the destination. To demonstrate this clearly, set up two terminals: one logged into the source instance (for example, 10.0.1.10) and one on the destination instance (10.10.2.20). Start a capture on both machines filtering specifically for the target port.

BASH

# On the destination host (10.10.2.20):
sudo tcpdump -nn -i any port 3306

# On the source host (10.0.1.10), in a separate terminal:
sudo tcpdump -nn -i any port 3306

With both captures running, attempt a simple TCP handshake from the source using nc (netcat) or curl:

BASH

nc -zvw3 10.10.2.20 3306

Now, compare the outputs from both terminals. This comparison reveals the exact location of the failure:

The Dead Path Signature: On the source host, you see multiple [SYN] packets sent with increasing backoff intervals (retransmissions). On the destination host, the capture stays completely blank. The packets are disappearing somewhere in the transit fabric before reaching the target subnet.
The Asymmetric Stateful Drop Signature: On the source host, you see repeated [SYN] packets. But on the destination host, you see the incoming [SYN] arrive, followed immediately by the destination host sending a [SYN, ACK]. However, that [SYN, ACK] never appears on the source host's packet capture. The outbound route from the destination back to the source is dropping the response.
The Local Host RPF Drop Signature: On the destination host, tcpdump shows the incoming [SYN], but the destination never emits a [SYN, ACK] at all, even though a service is actively listening on that port. This usually means the Linux kernel dropped the packet during ingress route verification.

Checking Kernel Reverse Path Forwarding

When the destination server receives packets but refuses to reply over a multi-homed or complex route, verify the host-level RPF configuration. The Linux kernel controls this behavior via sysctl flags under net.ipv4.conf. You can inspect the RPF settings across all interfaces with a single read-only command:

BASH

sysctl -a 2>/dev/null | grep '\.rp_filter'

The values represent three distinct modes:

0 (Disabled): No source address verification. The kernel processes the packet regardless of which interface it arrived on.
1 (Strict Mode): The incoming packet is dropped if the interface it arrived on is not the best reverse path to the source IP, according to the local routing table.
2 (Loose Mode): The incoming packet is accepted as long as the source IP is reachable through any interface on the system, even if it is not the interface the packet arrived through.

If your VPC architecture relies on asymmetric routing—for example, if ingress traffic comes through an internal peering attachment on eth1 but default egress traffic routes out through a centralized NAT gateway on eth0—strict mode (1) will silently discard valid incoming connections. Changing the interface to loose mode (2) allows the host to process asymmetric flows while still offering basic protection against entirely unroutable source addresses.

What Is Easy to Misunderstand

A common misunderstanding is assuming that ICMP ping tests confirm network symmetry. Cloud routing tables and security groups often handle ICMP differently than TCP. Furthermore, an ICMP echo request might follow a symmetric path if it matches a default route, while TCP traffic to a specific port might hit a policy-based route that steers it toward an inspection appliance. Another misconception is trusting single-ended traceroutes. When you run traceroute from host A to host B, you are only discovering the forward path of TTL-expired messages. The routers returning the ICMP Time Exceeded packets generate their own return paths back to host A. A clean traceroute in one direction tells you nothing about whether host B's TCP responses can follow that same path in reverse. Finally, support engineers often assume that if a network partition occurs inside a cloud provider's backbone, transit routes will automatically withdraw. In reality, SDN control planes can maintain static transit gateway attachments even when the underlying data plane spine between two availability zones or regions is impaired. The route table entry remains active, pointing traffic into a black hole or forcing partial failovers that break flow symmetry.

Testing the Behavior Safely

You can easily reproduce this troubleshooting workflow on two test VPS instances without touching production infrastructure. Spin up two small instances connected via an internal network. On the second instance, use iptables to simulate an asymmetric stateful drop:

BASH

# Simulate a stateful firewall dropping out-of-order replies on the test receiver
sudo iptables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK -j DROP

When you initiate a connection from the first instance, run tcpdump on both ends. You will observe the exact pattern discussed earlier: the client sends repeated SYNs, the receiver receives them, but the output rule drops the SYN-ACK before it hits the wire. Once you learn to recognize that signature in a safe environment, you will instantly identify asymmetric routing drops when real cross-VPC links misbehave.