Why Transparent Huge Pages Turn Redis Forks into Instant OOM Kills

A high-throughput Redis instance sitting at 22 GB Resident Set Size (RSS) on a 32 GB bare-metal node should, in theory, survive a routine background snapshot. Operating system folklore dictates that fork() is virtually instantaneous thanks to copy-on-write (COW) semantics: parent and child share read-only physical pages, and only modified data incurs physical memory duplication. Yet, infrastructure teams routinely watch these instances get abruptly executed by the Linux Out-Of-Memory (OOM) killer midway through an RDB save or replica synchronization. A technical teardown by OneUptime on Redis Copy-on-Write mechanics highlights the structural reality of how Redis relies on OS-level page sharing during BGSAVE and BGREWRITEAOF. While standard memory modeling assumes COW overhead scales proportionally with the net volume of modified data, this assumption disintegrates when Linux Transparent Huge Pages (THP) are active. Instead of paying a marginal penalty for altered records, THP forces a catastrophic 512-fold amplification on every modified memory page, turning routine disk dumps into instant process termination.

The Anatomy of the 512x COW Amplification

When Redis invokes fork() to initiate a snapshot, the Linux kernel does not duplicate user-space memory. It clones the parent's page directory entries, increments the reference count on the underlying physical page frames, and marks the Page Table Entries (PTEs) for both parent and child processes as read-only. When the Redis parent process subsequently receives a command that alters an existing key, the CPU attempts to write to a page marked read-only, triggering a minor page fault handled by do_wp_page(). Under the standard x86-64 memory architecture, memory is managed in 4 KB base page frames. When Redis modifies a 64-byte string inside a 4 KB page, the kernel's fault handler allocates a single 4 KB physical frame, copies the 4,096 bytes from the shared page, updates the parent's PTE to point to the new frame with read-write permissions, and clears the write-protect flag. The COW footprint of that write is exactly 4 KB. When Transparent Huge Pages (CONFIG_TRANSPARENT_HUGEPAGE) are enabled in always or madvise mode, the kernel coalesces contiguous 4 KB pages into 2 MB huge pages (512 base pages) to minimize Translation Lookaside Buffer (TLB) misses. In compute-heavy, read-dominated applications scanning contiguous arrays, this drastically reduces page-walk penalties. But Redis is an in-memory pointer soup: keys, values, dictionary buckets, and SDS strings are fragmented across heap space via jemalloc. When a single 64-byte key inside an active 2 MB huge page is mutated during a fork, the kernel cannot perform a partial 4 KB copy. It must allocate an entirely new 2 MB physical frame, copy all 2,097,152 bytes across the memory bus, and repoint the Page Middle Directory (PMD) entry. Updating 5,000 scattered keys across your dataset does not generate ~20 MB of COW overhead; it can allocate 10 GB of physical RAM within seconds. On a node with only 8 GB of head-room, the kernel runs out of unallocated frames, fails back-pressure allocation, invokes out_of_memory(), and sends SIGKILL directly to the Redis parent.

Latency Cliffs and Kernel Compaction Locks

The ballooning physical RSS is only half the crisis; the other half is the complete degradation of the Redis single-threaded event loop. Copying 4 KB of data in L1/L2 cache during a base-page fault takes roughly 200 to 500 nanoseconds—fast enough that Redis can handle high write volumes without blowing past client timeout thresholds. Copying a 2 MB frame requires fetching two full megabytes from system DRAM, copying it into newly mapped frames, and flushing the processor's local TLB entries. Under memory pressure, the penalty deepens: if the kernel cannot immediately find a contiguous 2 MB physical block of memory, it triggers direct compaction. The executing thread (Redis's primary event loop) stalls while the kernel scans zone lists, migrates surrounding pages to free up a contiguous block, and locks memory zones via spinlocks.

TEXT

[Redis Event Loop Thread]
   +-- Write to key "session:user:12908"
         +-- CPU Trigger: Page Fault (#PF, CR2=0x7f8a94200000)
               +-- Kernel: do_wp_page()
                     +-- alloc_hugepage_vma()  <-- Fails to find contiguous 2MB
                           +-- try_to_compact_pages()
                                 +-- Zone lock contention (Stall: 12ms - 80ms)

A write operation that usually executes in sub-microsecond time is suddenly forced to wait tens of milliseconds in kernel space. To external clients, Redis appears completely hung, queuing connections until TCP listen backlogs overflow and health checkers conclude the node has crashed.

Auditing the THP Footprint in Production

To diagnose whether an unexpected OOM was caused by huge page COW amplification or an actual organic dataset leak, inspect the process's Virtual Memory Areas (VMAs) via /proc. Redis reports basic COW metrics in its INFO logs, but the Linux VFS interface provides ground truth on how much memory is trapped in huge pages. You can observe this live during a fork by examining the smaps_rollup of the primary process:

BASH

# Inspect live THP and Private Dirty allocation on the Redis master
grep -E '^(Private_Dirty|AnonHugePages):' /proc/$(pgrep -o redis-server)/smaps_rollup

If AnonHugePages accounts for a substantial fraction of Private_Dirty while a snapshot child is running, your node is actively bleeding physical frames to 2 MB COW copies. If your memory monitoring system only watches used_memory reported by Redis internals, you will be blind to this disaster: used_memory tracks allocations made by the internal allocator (jemalloc), which remains oblivious to the OS-level duplication of physical pages caused by THP.

Hardening the Host

Eliminating this failure mode requires purging THP from the host operating system entirely. Disabling it within the application configuration is impossible because jemalloc historically interacts with Linux virtual memory via mmap() without explicitly passing MADV_NOHUGEPAGE to all heap allocations. To eliminate THP-induced COW spikes, disable both transparent huge page allocation and runtime defragmentation at the kernel level:

BASH

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Because system distributions routinely reset these sysfs nodes on reboot or via tuned profiles (such as tuned-adm), persist this configuration by appending transparent_hugepage=never to the kernel boot command line in /etc/default/grub, followed by running update-grub. Equally critical is the node's overcommit configuration. Setting sysctl vm.overcommit_memory=1 instructs the kernel to permit the initial fork() even if the address space allocation exceeds physical RAM, trusting that COW will not immediately duplicate the entire space. Once THP is set to never, COW returns to its predictable, 4 KB granularity. The parent process can continue serving mutative queries alongside active persistence workers without generating synthetic gigabytes of write-amplified memory pressure.