TCP Truncation at the Edge: Why Short Envoy Delayed Close Timeouts Corrupt Serialized Payloads

A high-throughput edge proxy should be an invisible conduit: bytes enter from an upstream cluster, travel through an event loop, and stream reliably down to clients across varying network topologies. Yet, in production environments moving large serialized payloads—such as multi-megabyte Protocol Buffer records, large JSON datasets, or bulk analytics chunks—edge connections frequently suffer from mysterious, intermittent truncation. Clients report malformed JSON syntax errors, incomplete Protobuf streams, or raw Connection reset by peer errors, even though upstream services report an unambiguous HTTP 200 with zero internal faults. This exact failure mode surfaced as a core dilemma in Envoy Proxy issue #6616, where engineers debated the delicate race condition between Envoy's delayed_close_timeout and the flushing of the connection write buffer. The problem exposes a fundamental tension in network systems: the boundary where user-space event loops hand data to the operating system's TCP stack, and what happens when application-level teardown timers run faster than physical packet transit across the internet.

Anatomy of Delayed Close and Socket Teardown

When Envoy finishes writing a downstream response on an HTTP connection scheduled for closure—such as an HTTP/1.1 request lacking keep-alive, a downstream stream reset, or an edge proxy enforcing maximum connection lifespans—it cannot simply call close() on the socket and walk away. TCP is full-duplex, and half-closing via shutdown(fd, SHUT_WR) sends a FIN segment to indicate the proxy has no more data to transmit. However, if the client sends spurious data immediately after, or if unread data remains in the proxy's socket receive buffer when user-space closes the file descriptor, the Linux kernel bypasses the four-way handshake entirely and emits a TCP RST. To shield downstream clients from these resets, Envoy implements a "delayed close" state. Once the proxy flushes its user-space buffers into the kernel socket, it initiates a timer governed by delayed_close_timeout (defaulting historically to 1 second) and registers the socket for read events. The proxy waits for the downstream client to consume the response data and transmit its own FIN, acknowledging the clean end of transmission. Trouble begins when operators tune delayed_close_timeout down to aggressive values like 20ms to 50ms. Infrastructure teams often do this to satisfy short health-check intervals from intermediate load balancers like Varnish, or out of fear of connection leaks under high concurrency. But on any connection spanning a public network, 20ms is barely enough time for a single round-trip, let alone draining a multi-megabyte buffer over a congested link.

The Kernel Write Queue vs. The Event Loop

To understand why aggressive timeouts corrupt serialized payloads, you have to follow where the bytes actually reside when the timer starts ticking. When Envoy executes writev() on a socket, user-space considers those bytes dispatched as soon as the system call returns successfully. At that instant, Envoy's application-level buffer watermark drops to zero, and the proxy arms the delayed_close_timeout.

YAML

# Risky edge configuration: aggressively truncates pending socket buffers
typed_config:
  "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
  delayed_close_timeout: 0.05s # 50ms is shorter than common WAN round-trip times

However, returning from writev() only proves that the bytes made it across the user-kernel boundary into the socket's write buffer (sk_write_queue). The kernel's TCP stack must still divide those bytes into Maximum Segment Size (MSS) units, serialize them through the network interface, and wait for cumulative TCP acknowledgments (ACKs) from the client.

TEXT

[ Envoy Event Loop ] -- writev() --> [ Kernel sk_write_queue (SO_SNDBUF) ]
        |                                       |
 Arms delayed_close_timeout (50ms)       Transmits packets over WAN (RTT ~70ms)
        |                                       |
 Timer expires!                                 |
 Calls close(fd) -----------------------------> ABORT! Sends TCP RST
                                         (In-flight payload destroyed)

If the client is separated from the edge proxy by a 70ms round-trip time (RTT) and experiencing TCP window throttling or packet loss, those serialized bytes linger in sk_write_queue. When the 50ms timer expires, Envoy's event loop executes its teardown handler and calls close(fd). Because unacknowledged data still resides in the kernel write buffer, the Linux kernel does not cleanly transmit a FIN; it terminates the Transmission Control Block (TCB) immediately and flings an active TCP RST downstream. The client's TCP stack receives the reset, discards any segments currently buffered in its own receive queue, and aborts the socket read. The application layer above it receives an incomplete payload right in the middle of parsing a critical JSON or Protobuf boundary.

The Fallacy of Disabling the Delayed Close Timer

Faced with truncated downloads, some teams choose the opposite extreme: disabling the delayed close timer altogether by setting delayed_close_timeout: 0s. This forces Envoy to completely skip the delayed-close state, immediately tearing down the socket structure as soon as user-space writes complete. Disabling the timer introduces two catastrophic failure modes. First, it re-opens the original race condition where any lagging upstream packet or trailing client byte triggers an immediate TCP RST. Second, it completely eliminates the proxy's defensive posture against slow or malicious clients. Without a delayed close timeout, an edge proxy relying on downstream peers to complete the four-way FIN handshake can be held hostage by clients that refuse to send their final FIN. Sockets linger in FIN_WAIT_2 or proxy connection tracking tables indefinitely, steadily consuming file descriptors and kernel slab allocations until the node exhausts its ephemeral connection budget. Neither an arbitrary sub-100ms timeout nor a completely disabled timer solves the physics of network buffering. As Alyssa Wilk and the Envoy maintainers identified during the resolution of issue #6616, the delayed close mechanism must account for both the event-loop scheduling latency and the physical time required to flush the kernel's transmission pipeline.

Calibrating Against Network Delay and Buffer Watermarks

Eliminating truncation requires tuning the proxy's operational boundaries to respect both the Bandwidth-Delay Product (BDP) and the Linux kernel's memory allocation limits. A connection moving payloads at rate B over a path with round-trip time RTT requires a minimal buffer capacity equal to BDP = B × RTT. If an edge proxy is serving a 10 MiB payload to a client with an RTT of 80ms over a 100 Mbps broadband link, the transmission pipeline takes over 800ms of sustained line-rate transmission just to drain the payload from the proxy host. To observe whether your edge nodes are prematurely cutting off sockets with un-ACKed data, inspect the socket write queues directly via socket statistics:

BASH

# Inspect downstream connections in FIN_WAIT states with lingering write queues
ss -tni 'sport = :443' state fin-wait-1

In the ss diagnostic output, monitor the Send-Q column alongside the internal TCP metrics wreq, bytes_acked, and rtt. If you observe sockets in FIN_WAIT_1 with a non-zero Send-Q that vanish abruptly accompanied by an increase in the kernel's TCPAbortOnClose counter (visible via nstat -z TcpExtTCPAbortOnClose), the proxy is aggressively closing sockets before the kernel can drain the network queue. For systems engineering teams running Envoy at the edge, apply these architectural rules: First, align delayed_close_timeout with reality. The timeout should never be set lower than your 99th percentile client RTT plus the worst-case time needed to clear an operational socket buffer. In practical edge deployments facing public internet traffic, delayed_close_timeout should remain at or above its default of 1s, and can be safely extended to 5s–10s on endpoints dedicated to streaming large assets. Second, manage backpressure in user-space before bytes hit the kernel. Configure Envoy's downstream buffer watermarks (initial_stream_window_size and connection_buffer_limits) to prevent the proxy from reading whole multi-megabyte payloads from fast upstreams into socket memory faster than the downstream client can acknowledge them. When downstream write buffers remain bounded, the volume of data subject to the delayed close drain race is kept to a strict, predictable minimum.