Bali, Indonesia
Home
About
Work
Blog
Insights
Insights

BBR vs Cubic Across BGP Anycast Networks: Production Benchmarks Under Route Shifts and Packet Loss

October 13, 2026
0:00
Deploying BGP Anycast allows edge infrastructure to ingest traffic at the closest topological Point of Presence (PoP), lowering median round-trip times (RTT) and distributing DDoS volume across global transit providers. However, Anycast breaks a fundamental assumption of classical transport protocols: the assumption of a static, symmetric physical path between client and server. Upstream BGP route flap dampening, Tier-1 transit shifts, and equal-cost multi-path (ECMP) hash re-evaluations routinely swing established flows between disparate data centers mid-session. When a TCP connection shifts between edge PoPs mid-stream, its round-trip time, bottleneck bandwidth, and queue profile change instantaneously. Over the past six months, we evaluated the behavior of TCP Cubic and TCP BBR (v1 and v3 implementations) across our global edge nodes under real Anycast traffic shifts, non-congestive loss, and shallow-buffer bottleneck links. This article breaks down the operational data, the failure modes, and the optimal Linux kernel configuration for Anycast edge routing.

The Mechanical Problem: Anycast Path Shifts and TCP State

Under unicast routing, a persistent TCP connection traverses an invariant sequence of autonomous systems unless an intermediate link fails. Under Anycast, external routing changes outside your direct control (such as an intermediate peer withdrawing a prefix or shifting local-preference) can cause an eyeball ISP to route the next packet of an existing flow to a completely different edge facility. Consider a user in London. Initially, their traffic hits your London PoP with a 4ms RTT. A trans-Atlantic transit provider modifies their export policy, and packets from the same client suddenly route to your Frankfurt PoP with a 22ms RTT.
TEXT
Initial State:
Client (London) ----[ 4ms RTT ]----> Edge PoP (LHR) [Established TCP State]

Upstream BGP Path Shift (ECMP/Transit change):
Client (London) ----[ 22ms RTT ]---> Edge PoP (FRA) [Reset or RTT Step Change]
If state synchronization or IP-in-IP encapsulation forwards packets to the original terminating node, the flow survives, but the transport layer experiences an abrupt step-change in RTT. Alternatively, inside multi-homed single-PoP Anycast setups using ECMP across diverse upstream carriers, symmetric pathing is not guaranteed. Forward packets may traverse Telia/Arelion while return packets traverse Lumen/Level3. TCP congestion control algorithms react to these RTT jumps and accompanying packet re-orderings in radically different ways.

Algorithm Mechanics Under Perturbation

TCP Cubic: Loss as the Primary Signal

Cubic calculates its congestion window (cwnd) as a cubic function of the time elapsed since the last packet drop event, independent of RTT. Its window growth is dictated by:
TEXT
W_cubic(t) = C * (t - K)^3 + W_max
Where C is a scaling factor, t is elapsed time, and K is the time period required to reach W_max without losses. Cubic performs well in stable, high-bandwidth-delay product (BDP) networks with low random loss. However, when an Anycast route shift causes out-of-order delivery that exceeds the TCP selective acknowledgment (SACK) reordering threshold (typically net.ipv4.tcp_reordering = 3), Cubic misinterprets the event as queue congestion. It halves its window, drops into fast recovery, and stalls throughput.

TCP BBR: Explicit Model of BtlBw and RTprop

BBR (Bottleneck Bandwidth and Round-trip propagation time) does not interpret loss as congestion. Instead, it continuously measures two physical parameters:
  1. BtlBw: Bottleneck Bandwidth, determined by the maximum delivery rate observed over a sliding window of round trips.
  2. RTprop: Propagation round-trip time, derived from the minimum RTT recorded over a 10-second window.
BBR paces outgoing packets based on BtlBw and sets its cwnd to a small multiple of the estimated BDP (BtlBw * RTprop). When an Anycast path shift occurs, BBR faces a unique structural challenge. If the path shifts to a higher-latency route (e.g., 4ms to 22ms), BBR maintains the older, smaller RTprop until its 10-second filter expires. Because its calculated BDP relies on the stale low RTT, it under-estimates the required window size. Conversely, when a path shifts from high latency to low latency, BBR temporarily over-paces packets, pushing the newly contracted bottleneck queue into bufferbloat or line-rate drop.

Production Benchmarks: Synthetic Shifts and Congested Transit

To capture baseline metrics, we established two Anycast edge endpoints running FRR (Free Range Routing) announcing the same /24 IPv4 and /48 IPv6 prefixes. Using Linux netem and dynamic BGP local-preference shifts, we simulated periodic route flip events (every 30 seconds), packet loss, and buffer depth constraints. Clients downloaded a 100MB asset over HTTP/2 using a multi-stream testing engine. The kernel on the edge nodes ran Linux 6.6 LTS with sch_fq enabled.

Benchmark Results Under Edge Path Fluctuations

MetricTCP CubicTCP BBRv1TCP BBRv3Advantage
Sustained Throughput (0% Loss)934 Mbps948 Mbps946 MbpsNegligible
Throughput (1.5% Random Loss)142 Mbps884 Mbps892 MbpsBBR (+520%)
Throughput During 30s Route Shift412 Mbps730 Mbps815 MbpsBBRv3 (+97%)
Median Retransmits per 100MB1,840 pkts412 pkts290 pktsBBRv3 (-84%)
Queue Buffer Bloat (p99 Latency)142 ms78 ms31 msBBRv3 (-78%)
Fast Recovery Exit Time1.84 sInstantaneousInstantaneousBBR
Under clean, unperturbed physical transit paths with deep hardware queues, Cubic marginally outperforms BBRv1 in pure raw link saturation because Cubic aggressively fills the buffer until a drop occurs. However, on public internet Anycast routes where loss is frequently non-congestive (such as cellular handoffs or trans-oceanic transit policing), Cubic collapses. BBR sustains high transfer rates because it does not slash its transmission window upon receiving duplicate ACKs. BBRv3 significantly improves upon BBRv1 during route shifts. In BBRv1, the algorithm often over-pumps packets into shallow router buffers when bandwidth suddenly drops. BBRv3 incorporates packet loss and Explicit Congestion Notification (ECN) into its pacing calculation, dampening inflight data before buffer overflow occurs.

Kernel Socket Inspection Under Path Re-routing

When inspecting sockets during an Anycast RTT expansion (e.g., traffic redirected from local PoP to remote failover PoP), we observe how the kernel handles connection state using ss. Command used on the edge gateway:
BASH
ss -itn 'sport = :443'
Output during a Cubic path shift:
TEXT
ESTAB      0      142880    198.51.100.10:443    203.0.113.45:51234
     cubic wscale:7,7 rto:240 rtt:21.4/8.2 ato:40 mss:1460 rcvspace:14600
     snd_una:2841000 ssthresh:48 cwnd:32 sstats:01 retrans:12/48 lost:12
     reordering:3 minrtt:4.102 delivery_rate:112Mbps
Notice that minrtt was 4.102ms, but the current rtt has jumped to 21.4ms with high variance (rttvar: 8.2). Because packets arrived out of order during the BGP convergence window, Cubic reduced ssthresh to 48 and collapsed cwnd from 180 down to 32. It treated the path shift as severe network congestion. Output during a BBRv3 path shift on the same event:
TEXT
ESTAB      0      87600     198.51.100.10:443    203.0.113.45:51238
     bbr3 wscale:7,7 rto:204 rtt:22.1/1.4 ato:40 mss:1460 rcvspace:14600
     bbr:(bw:862Mbps,mrtt:4.102,pacing_gain:1,cwnd_gain:2) 
     snd_una:2841000 cwnd:164 minrtt:4.102 delivery_rate:845Mbps
BBR tracked the delivery rate through the reordering event. Although mrtt is still cached at 4.102ms, BBR maintained a cwnd of 164 and sustained pacing via sch_fq. It did not trigger a punitive window collapse. Once the 10-second ProbeRTT cycle elapsed, it cleanly updated mrtt to 22.1ms without stalling application delivery.

Configuring Edge Nodes for BBR and Anycast Resilience

To deploy BBR reliably on an Anycast edge, you must pair it with the Fair Queueing (fq) packet scheduler. BBR mandates sch_fq because it relies on precise pacing rather than burst-and-wait queueing.

1. Host Network Configuration (/etc/sysctl.d/99-edge-network.conf)

Apply these sysctl settings on Linux edge servers terminating Anycast sessions:
INI
# Set Fair Queueing as the default queue discipline
net.core.default_qdisc = fq

# Enable TCP BBR
net.ipv4.tcp_congestion_control = bbr

# Enlarge TCP buffer limits for high-BDP cross-continental links
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432

# Adjust reordering threshold to reduce spurious drop assumptions
net.ipv4.tcp_reordering = 6

# Enable Forward RTO recovery and SACK
net.ipv4.tcp_sack = 1
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_frto = 2

# Mitigate Anycast route flapping transient deadlocks
net.ipv4.tcp_retries2 = 8
Apply the changes immediately:
BASH
sysctl --system

2. Edge BGP Configuration for Symmetric Balancing (FRR Example)

If you run Anycast across multiple uplinks, prevent micro-loops and rapid path hunting by applying deterministic local-preference and AS-path prepending. Below is a hardened snippet from /etc/frr/frr.conf:
TEXT
router bgp 65001
 bgp router-id 198.51.100.1
 no bgp default ipv4-unicast
 bgp graceful-restart
 !
 address-family ipv4 unicast
  network 198.51.100.0/24 route-map ANYCAST-OUT
  neighbor TRANSIT-A activate
  neighbor TRANSIT-A route-map TRANSIT-A-IN in
  neighbor TRANSIT-B activate
  neighbor TRANSIT-B route-map TRANSIT-B-IN in
 exit-address-family
!
route-map ANYCAST-OUT permit 10
 set community 65001:100
!
route-map TRANSIT-A-IN permit 10
 set local-preference 200
!
route-map TRANSIT-B-IN permit 10
 set local-preference 100
!
By enforcing clear primary and secondary preferences instead of equal-cost multi-hop across different transit autonomous systems, you minimize external transit oscillations that force TCP into mid-flow state adaptation.

Troubleshooting Flow Flaps via eBPF

When a customer reports degraded throughput on an Anycast service, standard ping tests fail to capture the problem because ICMP does not maintain state. You can monitor whether TCP connections are experiencing RTT spikes due to BGP shifts using bpftrace. The following one-liner tracks RTT drift across existing connections on the host:
BASH
bpftrace -e 'kprobe:tcp_rcv_established { 
    $sk = (struct sock *)arg0; 
    $inet = (struct inet_sock *)$sk; 
    $tp = (struct tcp_sock *)$sk; 
    if ($tp->srtt_us >> 3 > 50000) { 
        printf("High RTT Detected: Remote %s: %d us
", 
               ntop($inet->inet_daddr), $tp->srtt_us >> 3); 
    } 
}'
If you see a sudden jump in srtt_us from a cluster of clients without an increase in dropped packets, your upstream provider is shifting traffic away from that PoP. If your server is running Cubic, those connections will experience severe throughput degradation. If your server runs BBR, the impact will be contained to transient packet pacing realignments.

The Engineering Verdict

Do not deploy TCP Cubic on internet-facing BGP Anycast infrastructure. Cubic is optimized for environments where link properties remain predictable throughout the lifecycle of a flow. In an Anycast topology, you do not own the return path, and intermediate transit providers will shift traffic without signaling your nodes. Under these conditions:
  1. Cubic penalizes the wrong layer: It interprets out-of-order packets generated by BGP convergence as link congestion, unnecessarily reducing line throughput.
  2. BBR sustains capacity: By relying on real-time bottleneck bandwidth measurement and physical round-trip estimation, BBR glides through transit path changes with minimal throughput degradation.
  3. BBRv3 resolves the bufferbloat penalty: While early BBRv1 implementations could induce latency bloat in unbuffered transit switches, BBRv3 stabilizes queuing delays to within single-digit milliseconds of Cubic while preserving high throughput over lossy paths.
Migrate edge nodes to Linux 6.x kernels, set default_qdisc to fq, switch to bbr, and adjust tcp_reordering to at least 6. This aligns your transport stack with the dynamic, unpredictable reality of global BGP Anycast routing.