Taming Bcache Writeback Throttling: Diagnosing Block I/O Stalls Under Bursty Write Concurrency

When configuring tiered storage on Linux, pairing high-capacity mechanical drives with a fast NVMe caching tier through bcache seems like a pragmatic architecture. In theory, writes land immediately on flash in writeback mode, returning instant write acknowledgments before a background thread asynchronously streams dirty blocks to the backing spindles. In practice, however, storage systems subjected to unpredictable, bursty write concurrency often hit a wall. As benchmarked in Giuseppe Della Bianca's teardown on Fedora Magazine, mixed random read/write workloads can suddenly degenerate to a fraction of a megabyte per second—tumbling as low as 0.15 MiB/s on rotary arrays with write latencies spiking well beyond 170ms. This behavior is rarely caused by flash endurance degradation or raw hardware failure. Instead, it stems from how the Linux kernel's bcache subsystem arbitrates dirty buffer exhaustion. When foreground processes dump data onto the SSD faster than the rotational backing device can drain it, the internal rate controller panics. To protect the cache from filling completely, bcache forces foreground I/O to match the throughput of the underlying spinning disks. Understanding and mitigating these latency cliffs requires a deep look into bcache's Proportional-Integral-Derivative (PID) writeback controller.

The Mechanics of Bcache's PID Controller Collapse

Inside drivers/md/bcache/writeback.c, the kernel attempts to smoothly regulate the speed of background flushes using a discrete PID controller. The primary setpoint is /sys/block/bcache<N>/bcache/writeback_percent, which defaults to 10% of total cache capacity. If the ratio of dirty blocks to total cache blocks sits below this target, the writeback rate ramps down to avoid stealing I/O bandwidth from foreground reads. When dirty blocks exceed the target, the controller scales up the writeback rate. The calculation happens periodically, driven by writeback_rate_update_seconds (defaulting to 5 seconds). During high-concurrency bursts, several gigabytes of dirty data can land in the cache within hundreds of milliseconds. By the time the controller re-evaluates the system state 5 seconds later, the dirty percentage has dramatically overshot its setpoint. The error term spikes, driving the proportional and derivative components to their maximums. The controller cranks writeback_rate to astronomical values, flooding the backing device's elevator queue. Once the rotary backing device is saturated with writes, two crippling bottlenecks emerge. First, mechanical disk actuator arms suffer severe seek thrashing if the dirty blocks are non-contiguous. Second, and more critically, any foreground read that misses the cache must queue behind this avalanche of flush requests. When the dirty cache approaches capacity, bcache switches from asynchronous caching to synchronous backpressure, stalling the writing processes directly in the kernel block layer until dirty sectors drop back under safe limits.

Tuning Sysfs Controllers Against Windup and Thrashing

Recovering predictable write latency requires dampening the controller's oscillations and preventing the backing disks from descending into queue congestion. Default bcache sysfs parameters are notoriously slow to react and prone to aggressive overshooting. We can adjust the feedback loop by altering the proportional coefficient, the integral windup limit, and the sampling interval.

BASH

# Tighten controller reaction time to 1 second to catch bursts early
echo 1 > /sys/block/bcache0/bcache/writeback_rate_update_seconds

# Increase proportional term divisor to dampen violent rate swings
echo 60 > /sys/block/bcache0/bcache/writeback_rate_p_term_inverse

# Enforce a strict writeback floor to avoid spin-up lag during bursts
echo 4096 > /sys/block/bcache0/bcache/writeback_rate_minimum

Shortening writeback_rate_update_seconds prevents the error term from compounding unseen during a burst. Dampening the proportional term via writeback_rate_p_term_inverse (a larger denominator decreases gain) stops the controller from wildly over-correcting. Furthermore, setting writeback_rate_minimum ensures that the background thread steadily clears dirty sectors even during quiet periods, preserving precious headroom on the SSD for future write spikes. Another vital parameter is writeback_percent. While a 10% dirty threshold works well on small cache drives, dedicating 10% of a modern 2TB NVMe drive to dirty data means the controller allows up to 200GB of unwritten data to pool. Flushing 200GB to a 150 MiB/s rotational drive requires more than twenty minutes of sustained, saturated disk activity. For multi-terabyte flash caches, reducing writeback_percent down to 1% or 2% minimizes writeback debt while still providing adequate burst absorption.

Tracing Writeback Stall Bottlenecks with bpftrace

Diagnosing whether an I/O stall originates from filesystem barrier syncs or bcache writeback throttling is impossible with standard tools like iostat. Standard utilities aggregate metrics across the block device, blurring the distinction between SSD flash writes and backing disk flushes. Using dynamic kernel tracing via bpftrace lets you inspect the internal state transitions inside bcache directly at runtime.

TEXT

# bpftrace -e '
kprobe:bcache_writeback_rate_calc {
    $dc = (struct cached_dev *)arg0;
    printf("dirty: %llu sectors, rate: %llu sector/s
",
           $dc->sectors_dirty.counter,
           $dc->writeback_rate.rate);
}'

Observing this trace during burst workloads exposes whether your controller is oscillating between extremes (e.g., dropping to 1 sector/sec and then suddenly jumping to hundreds of thousands of sectors/sec). If writeback_rate spikes while dirty sectors are barely above the threshold, your proportional gain is too aggressive. Conversely, if dirty sectors climb continuously without a proportional bump in the writeback rate, the backing disk is physically saturated, and your storage tier requires explicit I/O cgroup throttling on foreground writers to prevent catastrophic cache fill-up. Architecting stable hybrid arrays requires abandoning the assumption that flash caches can mask slow backing disks indefinitely. By dampening the bcache PID controller, shortening sampling windows, and rightsizing dirty limits to match the physical drain rate of the rotational hardware, you transform jagged, stall-ridden I/O profiles into deterministic, flat latency curves.