OOMKilled exit codes during peak morning ingress. The baffling part was not the death itself, but the application telemetry: the Go runtime reported heap allocations hovering comfortably at 280 MiB against an 800 MiB container limit. Memory profilers, runtime garbage collection logs, and /proc/[pid]/status resident set size (RSS) metrics showed no memory leak. Yet, the cgroup event log recorded an unambiguous kernel verdict: memory.events incremented oom_kill and the container was liquidated.
The root cause lay in a silent architectural collision between TCP receive window auto-tuning and the unified hierarchy of Linux cgroups v2. Under cgroups v1, network socket buffers (sk_buff) were historically tracked under the optional and frequently disabled memory.kmem.tcp.limit_in_bytes. Under cgroups v2, however, all kernel allocations associated with network sockets are directly charged against the container's unified memory.current balance.
The Mechanics of Silent Socket Charging
sk_buff wrapper and assigns the payload to the receiving socket's queue. In the kernel source (net/core/sock.c), whenever incoming payload demands memory expansion, the network subsystem invokes sk_mem_raise_allocated() to verify whether the socket can expand its buffer budget.
Inside cgroups v2, this call paths through mem_cgroup_charge_skmem(). Unlike user-space anonymous memory, which grows deterministically through brk or mmap, socket memory expands inside the network softirq context on behalf of remote peers. If your host kernel runs with standard distributions defaults for net.ipv4.tcp_rmem—typically 4096 131072 6291456 (representing minimum, default, and maximum receive buffer sizes in bytes)—a single TCP socket is permitted to inflate its receive buffer up to roughly 6 MiB to maximize the bandwidth-delay product (BDP).
BASH
# Inspecting kernel receive buffer boundaries (min, default, max)
sysctl net.ipv4.tcp_rmem
# net.ipv4.tcp_rmem = 4096 131072 6291456sk_rmem_alloc) begin buffering. At 4 MiB per socket across 150 connections, the socket buffers alone demand 600 MiB of kernel memory. To the kernel's memory controller, there is no distinction between socket buffers and application heap: both draw from memory.current. When the sum crosses memory.max, the cgroup OOM killer invokes out_of_memory(), targets the largest user-space consumer in the hierarchy, and kills the primary process.
Why Application Runtimes Remain Oblivious
mm_struct, and heap allocators like jemalloc or Go's mcache. Kernel socket buffers live entirely outside user-space addressable memory. They reside in slab caches and kernel page allocations tracked by the network stack. Consequently, profiling endpoints such as pprof or JVM JMX metrics report pristine heap health while the container is moments away from annihilation.
To diagnose this discrepancy in production, you must query the kernel cgroup interface directly. Inspecting /sys/fs/cgroup/<group>/memory.stat reveals the breakout of memory usage across categories. In an incident driven by socket inflation, anon and file remain low, while the sock entry climbs to hundreds of megabytes.
BASH
# Checking memory.stat inside the container or from the host slice
cat /sys/fs/cgroup/memory.stat | grep -E '^(anon|file|sock) '
# anon 293601280
# file 41943040
# sock 512753664ss -t -i shows sockets with inflated receive queues and enlarged receive windows (rb values reaching multi-megabyte thresholds). The kernel is aggressively doing what it was designed to do: widening the TCP window to prevent packet drops on high-throughput connections, completely blind to the fact that it is cannibalizing the container's narrow cgroup headroom.
Re-architecting Socket Buffers for Containerized Workloads
net.ipv4.tcp_rmem inside the pod via an init container or sysctl configuration, you cap the maximum theoretical buffer a stalled connection can consume:
BASH
# Clamp max receive buffer to 512 KiB per socket inside the namespace
sysctl -w net.ipv4.tcp_rmem='4096 87380 524288'SO_RCVBUF via setsockopt during socket initialization. However, setting SO_RCVBUF manually disables dynamic TCP window sizing altogether, freezing the buffer at twice the requested value (to account for sk_buff overhead). For services operating over variable wide-area networks (WAN), hardcoding SO_RCVBUF can severely restrict throughput across high-latency links.
A more sustainable architectural pattern combines modest tcp_rmem clamping with strict transport-layer backpressure. Application proxies must stop reading from upstream sockets only when internal processing buffers are genuinely full, and they should enforce aggressive read timeouts (SO_RCVTIMEO) on clients that flood data while the application waits for upstream services. Combining proactive application backpressure with continuous monitoring of memory.stat:sock ensures that TCP auto-tuning accelerates your data pipeline rather than serving as an invisible trigger for the container OOM killer."
],
"uncoveredConcepts": [
{
"concept": "sk_mem_raise_allocated kernel tracing",
"suggestedTitle": "Tracing Kernel Socket Buffer Growth with eBPF: From sk_buff to memcg Limits",
"tag": "eBPF"
},
{
"concept": "Bandwidth-Delay Product (BDP) in containerized networks",
"suggestedTitle": "TCP Window Auto-Tuning in Kubernetes: Balancing BDP Against Container Memory Quotas",
"tag": "Networking"
},
{
"concept": "SO_RCVBUF vs dynamic TCP window scaling",
"suggestedTitle": "The Cost of Manual Socket Buffer Sizing: Why SO_RCVBUF Breaks TCP Window Scaling",
"tag": "Networking"
}
]
}