The Kernel Flush Trap: Why cgroup blkio Bypasses Caused NVMe Node Stalls

A high-throughput batch worker running inside a production container managed to lock up an entire bare-metal NVMe node, despite having aggressive blkio bandwidth ceilings enforced. To external monitoring, the offending pod appeared completely benign: its reported block I/O bandwidth sat well below its configured 150 MB/s throttle. Yet, co-located services—including the local kubelet and systemd-journald—fell into deep kernel uninterruptible sleep (D state), heartbeats expired, and the node dropped out of the cluster. When we attached kernel tracers to inspect the NVMe submission queues, we found the drive saturated with hundreds of thousands of dirty page writebacks originating not from the container's cgroup, but from global kernel background flusher threads running in the root context.

The Split-Brain Between blkio and the Page Cache

The root cause lies in how cgroup v1 handles buffered I/O. In Linux systems running cgroup v1, the blkio subsystem and the memory subsystem exist as completely separate hierarchies. When an application calls write() on a regular file without the O_DIRECT flag, it does not write straight to the block device. It copies bytes into the page cache, marks those pages as dirty, and immediately returns success to userspace. At this moment, the write is accounted for by the memory cgroup as cache usage, but the blkio controller knows nothing about it. Sometime later, kernel flusher threads (wb_workfn running on system kworker threads) wake up to flush those dirty pages out to storage. In cgroup v1, because the flusher runs in the context of the root kernel task, the block I/O request is dispatched to the storage driver without any association to the container that originally dirtied the memory. The cgroup v1 blkio.throttle.write_bps_device limits are simply bypassed. The NVMe controller receives an unthrottled torrent of write requests submitted directly through kernel threads, starving the shallow submission queues and inducing latency spikes from tens of microseconds to hundreds of milliseconds.

The Cascade into Global Dirty Page Throttling

Once writeback flushes bypass cgroup throttling, the second failure mode triggers inside the kernel's virtual memory subsystem. As the container rapidly consumes memory by dirtying file buffers, the global dirty memory footprint climbs toward the system watermarks set by vm.dirty_background_ratio and vm.dirty_ratio. When total dirty pages cross the threshold where the background flushers cannot keep pace with incoming writes, the kernel invokes balance_dirty_pages(). This function forces any process attempting a write—regardless of which cgroup it lives in—to pause and synchronously perform writeback before it is permitted to allocate more pages. At this stage, lightweight control plane daemons like kubelet, containerd, or administrative SSH sessions trying to write to /var/log or ephemeral state directories get trapped in synchronous disk writes against an already saturated NVMe queue. What began as an unthrottled rogue container effectively seizes the entire memory bus and block storage interface, creating a node-level deadlock while local metrics show zero throttle violations on the culprit.

BASH

# Inspecting kernel dirty page balance stalls with bpftrace
bpftrace -e '
  kprobe:balance_dirty_pages { 
    @[comm] = count(); 
  }
'

Running this trace during an incident instantly exposes the damage: vital host daemons like sshd and dockerd show up alongside the batch workload, stalled in the dirty page throttle loop because the host cannot isolate the buffered churn.

Why Unified cgroup v2 Writeback Fixes Attribution

Resolving this architectural flaw required a fundamental redesign of the kernel's cgroup hierarchy. Under cgroup v2, the memory and io controllers are unified into a single tree. This enables cgroup-aware writeback (CONFIG_CGROUP_WRITEBACK), an internal mechanism that establishes direct lineage between cached pages and the block controller. When a container dirties a page in cgroup v2, the struct page pointer retains an explicit association with its originating mem_cgroup. When flusher threads construct struct bio requests to submit pages to the NVMe driver, the kernel inspects the page ownership and attaches the matching io_cgroup context using wbc_attach_and_unlock_inode(). The block layer then enforces io.max and io.weight limits on the actual writeback I/O, throttling the flusher if the originating cgroup has exceeded its target rate, and kicking backpressure straight into the offending container's write loop before the global page cache is overwhelmed.

INI

# Verify unified cgroup v2 writeback hierarchy on modern runtimes
# /sys/fs/cgroup/<slice>/io.stat must track writeback explicitly
writes=14209420 bytes=72740864000

Practical Hardening for Storage-Heavy Nodes

Migrating bare-metal hosts to full unified cgroup v2 hierarchy (by booting with systemd.unified_cgroup_hierarchy=1) is the mandatory baseline for stable multi-tenant I/O. Without it, configuring storage limits on buffered workloads is an illusion. Once on cgroup v2, container engines can reliably throttle both raw throughput (io.max) and proportional queue access (io.weight) without rogue kworker bypasses. Beyond the cgroup migration, operating NVMe drives under heavy write pressure requires tightening the dirty memory window. Default Linux distributions often ship with vm.dirty_ratio=20 and vm.dirty_background_ratio=10. On a server with 256 GB of RAM, a 20% limit allows over 50 GB of unwritten data to sit in flight before synchronous throttling begins—far too much data for an NVMe controller to digest without inducing tail-latency cliffs. Lowering vm.dirty_background_bytes to 268435456 (256 MB) and vm.dirty_bytes to 1073741824 (1 GB) forces the kernel to stream smaller, continuous writes to disk rather than bursting massive gigabyte-scale flushes that starve system-critical heartbeats.