Cgroup v2 Socket Accounting Pitfalls: How tcp_collapse Froze Our DoT Resolvers

During a routine traffic failover across our regional edge, our DNS-over-TLS (DoT) resolver fleet suddenly hit an operational cliff. Within ninety seconds of taking on an extra forty thousand concurrent TLS handshakes, p99 latency shot from 11 milliseconds to over 4,500 milliseconds. Handshakes were stalling mid-negotiation, and downstream client queries began timing out en masse. The initial telemetry was completely counterintuitive. The host machines had dozens of gigabytes of free RAM, CPU utilization sat well below 45%, and the kernel had not fired a single Out-Of-Memory (OOM) kill. The resolver processes were not crashing; their epoll event loops were simply starving. When we dropped into perf top on an affected edge node, the user-space resolver binaries were nowhere near the top of CPU accounting. Instead, kernel cycles were entirely consumed by tcp_collapse and tcp_prune_queue inside softirq context.

The Hidden Mechanics of Socket Memory Accounting

To understand why an epoll thread freezes under memory pressure, one must look at how the Linux network stack accounts for incoming TCP payloads. Every network packet arriving on an interface is wrapped in a kernel metadata structure called sk_buff. The memory footprint of this structure is not just the raw byte length of the incoming payload; it is represented by sk_buff->truesize, which includes the data buffer, alignment padding, and the sk_buff control structure itself. A 64-byte TCP packet can easily consume upwards of 1,500 to 2,048 bytes of actual slab and page memory. Under legacy cgroups v1, network socket memory was often ignored or segregated into the optional net_prio and memory.kmem.tcp.limit_in_bytes controllers, which most production container runtimes left disabled. However, cgroups v2 fundamentally changed this paradigm. Under unified hierarchy accounting, network socket buffers (memory.sock) are charged directly against the container's primary memory.current and evaluated against memory.max.

TEXT

Container memory.current
+-- Page Cache (File I/O)
+-- Anonymous Memory (Resolver Heap, TLS Sessions)
+-- memory.sock (sk_buff allocations via tcp_try_rmem_schedule)
      ^
      +-- Must stay below cgroup memory.max

When a containerized process receives a surge of traffic, the kernel calls tcp_try_rmem_schedule() to verify that the socket's receive buffer can expand. If this allocation pushes the container over memory.max, the kernel refuses to grow the socket buffer. Instead of immediately dropping the packet or invoking the OOM killer—which only triggers when user-space page allocations fail under direct reclaim—the TCP stack attempts to survive by compacting existing in-flight buffers.

Inside tcp_prune_queue and tcp_collapse

When tcp_try_rmem_schedule() returns false, the socket enters emergency queue pruning via tcp_prune_queue(). The kernel inspects the socket's out-of-order queue and receive queue to recover memory without dropping data on the floor. First, out-of-order packets are discarded to free immediate space. If memory pressure persists, the kernel executes tcp_collapse(). This function traverses the socket's linear receive queue, allocates a fresh, contiguous buffer, and copies the payloads from dozens of fragmented sk_buffs into a single contiguous chunk. By consolidating these fragments, the kernel discards the redundant sk_buff overhead headers, shrinking the truesize back toward the actual payload size.

BASH

# Check if your systems are silently burning CPU cycles collapsing buffers
nstat -az TcpExtTCPRcvCollapsed TcpExtTCPPruneDrop

While mathematically elegant, running tcp_collapse() under high concurrency introduces crippling latencies. The compaction loop runs with the socket lock held. If thousands of concurrent DoT connections cross their cgroup memory allocations at once, the softirq handler loops through linked lists of thousands of packets, executing memory copies (memcpy) inside interrupt and system call contexts. Because the socket lock is held during this compaction, any user-space call to epoll_wait or read on that file descriptor stalls waiting for the lock. The user-space event loop misses subsequent epoll wakeups, incoming TCP window advertisements drop to zero (win 0), and the entire resolver pipeline stalls.

Why DNS-over-TLS Amplifies the Problem

Standard UDP DNS queries are stateless and transient: a 60-byte request arrives, generates an immediate response, and releases its socket buffer instantly. DNS-over-TLS (RFC 7858), running over TCP port 853, alters the operational memory model entirely. First, TLS handshakes introduce multi-kilobyte bursts. A ClientHello accompanied by large cipher suites and SNI extensions often splits across multiple MTU frames. The server must buffer these frames until full TLS records can be reconstituted and decrypted by userland libraries. During connection setup, each socket holds several fragmented sk_buff allocations simultaneously. Second, default Linux sysctl parameters are designed for bulk throughput on modern high-bandwidth servers, not containerized micro-transactions. Consider the default TCP receive memory boundaries:

TEXT

net.ipv4.tcp_rmem = 4096  131072  6291456

The third value allows a single unconstrained TCP socket to claim up to 6 MB of receive buffer space under high bandwidth-delay product conditions. In a container configured with a 4 GB memory.max hosting 15,000 active DoT client connections, the mathematical upper bound for socket allocations far exceeds the container limit. When an edge traffic spike lands, the sockets naturally attempt to scale their receive windows, instantly slamming into memory.max. Instead of gracefully shedding load via standard TCP drops, the cgroup forced the entire pool of sockets into continuous, concurrent tcp_collapse passes. The system entered an unrecoverable thrashing state: softirq workers held socket locks to compress buffers, user-space was locked out from draining the sockets, and clients retransmitted their packets, worsening the queue backlogs.

Hardening the Socket and Cgroup Boundary

Resolving this architectural mismatch requires coordinating host-level TCP buffer ceilings with container-level cgroup boundaries so that memory pressure never triggers catastrophic kernel compaction loops.

1. Clamp sysctl tcp_rmem for Containerized Handshake Gateways

For a DoT resolver or TLS termination proxy, sockets carry small RPC-style messages rather than gigabyte bulk file transfers. Sockets do not need multi-megabyte window buffers. By clamping the upper boundary of tcp_rmem, you prevent individual sockets from inflating their sk_buff footprint:

BASH

# /etc/sysctl.d/99-dot-performance.conf
net.ipv4.tcp_rmem = 4096 16384 262144
net.ipv4.tcp_wmem = 4096 16384 262144

Setting the ceiling to 256 KB caps individual socket allocation while still providing more than enough room for full TLS 1.3 certificate chains and batched DNS pipelines.

2. Configure Cgroup memory.high as a Pressure Bleed

Never allow memory.sock to hit memory.max directly. In cgroups v2, memory.high acts as an asynchronous throttling throttle rather than a hard kill boundary. When usage crosses memory.high, the kernel proactively slows down allocating processes by injecting small delays in the return path to user space, giving background direct reclaim and memory management threads time to breathe before tcp_prune_queue is forced into synchronous execution. Ensure that memory.high is set to approximately 80% to 85% of memory.max. This leaves an essential buffer cushion specifically for network packet ingress during transient spikes.

3. Application-Level Socket Buffer Pinning

If tuning host sysctl parameters is impossible due to multi-tenant workloads sharing the same bare-metal kernel, clamp socket allocations directly in the DoT server daemon configuration or via code. Setting SO_RCVBUF via setsockopt forces the kernel to disable autotuning and hard-limits the receive queue allocation per connection, completely neutralizing the container-wide tcp_collapse avalanche. Operating edge TLS resolvers inside cgroups v2 requires recognizing that network buffers are no longer isolated kernel freebies. When memory limits are drawn, the networking stack must be budgeted with the same defensive boundaries applied to the heap.