Retiring Active Data Guard: Multi-Raft Heartbeat Overhead and Replication Unit Design in Oracle 23ai

When Oracle announced native Raft consensus replication for Globally Distributed Database in Oracle Database 23ai, the enterprise database world breathed a collective sigh of relief. For years, deploying sharded architectures on Oracle meant managing monolithic pairs of physical primary and standby nodes per shard using Active Data Guard (ADG) and complex Fast-Start Failover (FSFO) observer topologies. Moving to declarative consensus replication—where shards are divided into fine-grained Replication Units (RUs) maintaining quorum across three or five replicas—promises zero-data-loss failover without external broker orchestration. Yet, transitioning from monolithic physical replication to a distributed Multi-Raft state machine changes the physical network profile of your database hosts. Instead of a single, continuous, sequential TCP stream of database redo log buffers managed by an LNS (Log Network Server) process pushing to a remote RFS (Remote File Server), Multi-Raft distributes shards into dozens or hundreds of independent consensus groups. If engineered without accounting for kernel timer granularity and network interrupt handling, the aggregate heartbeat overhead of co-located consensus state machines will rapidly exhaust OS socket buffers and degrade transaction latency.

The Architectural Pivot: From Redo Streams to Replication Units

In traditional Active Data Guard, replication operates at the physical database boundary. An entire pluggable or container database streams its block changes sequentially over a point-to-point network link. While rock solid, this monolithic approach presents severe operational friction in distributed systems: if a single shard host experiences transient packet loss or storage latency, failing over that shard requires complex state machine negotiation through a centralized cluster arbiter or observer node. Oracle 23ai replaces this coarse model with Replication Units. An RU is a logical collection of table partition chunks that share an identical replication topology across storage shards. Rather than designating an entire bare-metal or virtual host as an active primary or passive standby, each shard hosts a mosaic of leaders and followers belonging to distinct RUs. When a shard fails, only the RUs where that host held leadership execute an automated leader election, settling quorums in sub-second timeframes using standard majority rules (Q = ⌊N/2⌋ + 1).

SQL

-- Verifying Replication Unit (RU) placement and status in 23ai
SELECT ru_id, chunk_id, shard_name, raft_role, raft_term 
FROM v$gdd_replication_unit_members 
ORDER BY ru_id, shard_name;

However, this granularity shifts the replication bottleneck from raw disk I/O serialization to operating system network scheduling. Where ADG maintained one persistent, high-throughput TCP connection between two database endpoints, a 23ai node hosting 200 RUs now engages in multiple independent consensus protocols concurrently, each emitting periodic heartbeats, tracking terms, and persisting individual index commit states.

The Multi-Raft Heartbeat Explosion

Under the Raft protocol, a leader must issue periodic empty AppendEntries RPCs to all followers to prevent election timeouts from triggering spurious re-elections. In a typical low-latency datacenter deployment, the heartbeat interval is configured aggressively—often between 50ms and 150ms—with election timeouts set to 500ms to achieve sub-second failover guarantees. When a single physical server hosts hundreds of RUs, this heartbeat traffic scales multiplicatively. Consider a conservative cluster with 64 physical shards, where each shard manages 128 RUs replicated across three nodes (N = 3). Each shard host participates in hundreds of consensus relationships with peer shards. If each RU leader fires an un-coalesced AppendEntries packet every 100ms, a single database host can generate thousands of individual UDP or TCP network transmissions per second purely to signal state liveness, entirely independent of user transactional workloads. At the Linux kernel level, this creates a distinct performance pathology: high-frequency timer interrupts (hrtimers) and softirq storms (NET_RX_SOFTIRQ). When consensus state machines run in isolated user-space threads or background database worker processes without transport multiplexing, the CPU incurs non-trivial context-switching overhead. The kernel continuously services timer expirations, pushes microscopic 64-byte payload frames onto interface TX rings, and awakens peer network stacks. The resulting latency jitter directly threatens the very stability Raft is meant to ensure: if thread starvation or socket lock contention delays a heartbeat past the election window, healthy followers prematurely trigger election campaigns, leading to disruptive term increments and split-brain recovery storms.

Log Slicing and Transport Pipeline Multiplexing

To prevent Multi-Raft consensus from choking Linux network queues, the replication engine must decouple consensus group logic from physical transport sockets. A naive implementation assigns distinct socket file descriptors to each RU, which quickly saturates the kernel's epoll sets and creates excessive TCP control block memory pressure. Modern Multi-Raft implementations bypass this by utilizing a multiplexed transport daemon. Instead of each RU emitting an isolated network packet per tick, the host engine aggregates outbound heartbeats targeted at the same destination node into a single vectorized network frame. When multiple RU leaders on Node A need to heartbeat their respective peer replicas on Node B, the outbound scheduler coalesces these checks into a unified batched payload.

TEXT

+-------------------------------------------------------------+
|                  Shard Node (Host A)                        |
|  [RU 1 Leader]    [RU 2 Leader]   ...   [RU N Leader]       |
|         \               |                /                  |
|          +--------------+---------------+                   |
|                         |                                   |
|           [Batched Transport Multiplexer]                   |
+-------------------------+-----------------------------------+
                          | Single Coalesced Stream (TCP/RDMA)
                          v
+-------------------------+-----------------------------------+
|                  Shard Node (Host B)                        |
|          [Transport Frame Demultiplexer]                    |
|         /               |                \                  |
|  [RU 1 Follower]  [RU 2 Follower] ...   [RU N Follower]     |
+-------------------------------------------------------------+

Furthermore, write-ahead log (WAL) persistence requires careful alignment. In ADG, redo logs are streamed and flushed sequentially to physical standby control and redo files. In 23ai Multi-Raft, every RU executes its own log append path. If each RU attempts to issue independent synchronous fdatasync() calls to underlying NVMe namespaces for every transactional append, the storage controller faces queue contention and write amplification. The consensus log layer must coordinate with Oracle's Log Writer (LGWR) process to perform group commits that span multiple RUs simultaneously, committing an entire batch of consensus index entries in a single I/O barrier.

Linux System Hardening for Low-Jitter Multi-Raft

Deploying 23ai with fine-grained Raft replication requires deliberate operating system and network stack tuning to shield database worker threads from jitter. You cannot rely on default network socket buffers when hundreds of micro-transactions and heartbeats traverse the stack. First, isolate database consensus workers from general host processes and eliminate softirq imbalances across CPU cores. Ensure that Receive Packet Steering (RPS) and Receive Flow Steering (RFS) distribute consensus traffic across cores that do not host latency-sensitive transaction processing engines:

BASH

# Increase TCP receive and transmit buffer memory for dense RPC traffic
sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'

# Enable low-latency busy polling to reduce softirq context-switch delays
sysctl -w net.core.busy_poll=50
sysctl -w net.core.busy_read=50

Setting busy_poll parameters allows socket reads to poll the network device queue directly when waiting for incoming Raft votes or append acks, substantially reducing context-switch delays and timer variance at the expense of slight CPU utilization. Additionally, watch the tcpretrans metrics closely; in a Multi-Raft deployment, even a 0.5% packet loss rate can trigger cascades of follower timeouts across dozens of concurrent RUs, causing rolling performance degradation across the cluster.

The Engineering Trade-off: Operational Freedom vs Infrastructure Overhead

Retiring Active Data Guard in favor of Oracle 23ai's Multi-Raft replication represents a major paradigm evolution for mission-critical relational infrastructure. The elimination of brittle external failover scripts, complex standby broker configurations, and long recovery time objectives (RTO) significantly improves day-two operational reliability. However, distributed consensus is not a free abstraction. It trades the straightforward, single-stream sequential I/O of traditional physical replication for an intricate mesh of timer loops, batched network RPCs, and granular lock contexts. Production success with 23ai Globally Distributed Database requires sizing replication chunk topologies conservatively: avoid the temptation to create thousands of micro-RUs where a few dozen well-balanced units will suffice. High availability is no longer just about storage replication fidelity—it is dictated by your system's ability to schedule and transport thousands of distributed heartbeats without dropping a single packet.