Titik Kehancuran: Amplifikasi I/O dan Socket Starvation
- Random Read Amplification: Storage engine melakukan traversal graph commit yang tersebar pada ribuan potongan log kecil, menyebabkan seek penalty tinggi dan cache eviction agresif pada buffer pool database metadata.
- TCP Socket Buffer Bloat: Daemon aplikasi membaca potongan objek ke user-space buffer lalu menulisnya kembali ke socket via syscall 'write()'. Saat interface jaringan mengalami micro-burst congestion, buffer kirim socket ('SO_SNDBUF') lekas penuh. Kernel memblokir thread worker atau memaksa alokasi 'sk_buff' yang memicu memory pressure ('slub/slab exhaustion').
TEXT
+-------------------------------------------------------------+
| Aplikasi Storage Engine |
+-------------------------------------------------------------+
| (1) Metadata Lookup | (3) Object Stream
v v
+-----------------------+ +----------------------+
| Append-Only Log Index | | Block Storage Engine |
| (LSM Tree / RocksDB) | | (Pack / Blob Chunks) |
+-----------------------+ +----------------------+
|
+-----------------------------------+
| (Direct I/O via NVMe)
v
+-------------------------------------------------------------+
| Linux Kernel VFS |
+-------------------------------------------------------------+
|
| ZERO-COPY PIPELINE: splice() / io_uring
v
+-------------------------------------------------------------+
| Network Subsystem (TCP Sockets) |
| [ sk_buff Queue ] -> [ qdisc / FQ ] -> [ NIC Ring TX ] |
+-------------------------------------------------------------+Anatomi Subsistem Jaringan: Manajemen sk_buff dan TCP Window
TEXT
+-------------------------------------------------------------------------+
| Struktur Memori sk_buff |
+-------------------------------------------------------------------------+
| struct sk_buff | struct skb_shared_info | Payload Data (Frag Array) |
| (Overhead) | (Paging Metadata) | (Direct Memory/Page Pointers) |
+-------------------------------------------------------------------------+
|<--------- Alokasi SLAB Kernel -------->|<----- Transferred via DMA ---->|Evaluasi Mekanisme Zero-Copy Storage ke Socket
| Metode Transfer | Mekanisme Data Path | Context Switches / I/O | Memory Copy Overhead | Resiliensi Backpressure |
|---|---|---|---|---|
| read() / write() | Disk -> User Space -> Socket | 2 syscalls (4 switches) | 2x Copy (Disk->RAM, RAM->Sock) | Rendah (Menghabiskan RAM aplikasi) |
| sendfile(2) | Page Cache -> Socket TX Ring | 1 syscall (2 switches) | 0x (Hanya ring descriptor) | Sedang (Terkunci pada descriptor file) |
| splice(2) | Pipe Buffer -> Socket Buffer | 1 syscall (2 switches) | 0x (Kernel Pipe Pages) | Tinggi (Buffer ukuran pipa terkendali) |
| io_uring | Asynchronous Ring Submission | 0 syscall (pada mode SQPOLL) | 0x (Registered Direct Buffers) | Maksimal (Fully Non-blocking Ring) |
Konfigurasi Tuning Kernel untuk High-Throughput Log Streaming
INI
# /etc/sysctl.d/99-storage-network-engine.conf
# Tingkatkan batas total memori jaringan TCP (dihitung dalam unit pages 4096 bytes)
# Format: min pressure max
net.ipv4.tcp_mem = 4194304 8388608 16777216
# Tuning buffer baca TCP (rmem) dan tulis TCP (wmem)
# Format: min default max (16MB max buffer untuk koneksi latensi menengah-tinggi)
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
# Matikan TCP slow start setelah idle agar throughput transmisi commit tetap stabil
net.ipv4.tcp_slow_start_after_idle = 0
# Wajib aktifkan TCP Window Scaling (RFC 1323)
net.ipv4.tcp_window_scaling = 1
# Gunakan BBRv3 sebagai algoritma congestion control untuk menjaga throughput
# tanpa memicu bufferbloat di queue switch rack
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# Perluas antrean backlog socket untuk menangani lonjakan handshake serentak
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 100000
# Optimasi TCP Selective Acknowledgment
net.ipv4.tcp_sack = 1
net.ipv4.tcp_dsack = 1Penerapan Direct Zero-Copy Streaming pada Daemon Engine
C
#define _GNU_SOURCE
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <sys/socket.h>
/*
* Mengalirkan chunk objek database langsung dari storage fd ke socket client
* tanpa menyalin byte data ke user-space buffer memory.
*/
ssize_t stream_blob_zero_copy(int storage_fd, off_t *offset, int client_socket_fd, size_t length) {
int pipe_fds[2];
ssize_t total_transferred = 0;
// Buat kernel pipe internal sementara
if (pipe2(pipe_fds, O_NONBLOCK) == -1) {
return -1;
}
while (total_transferred < length) {
size_t to_transfer = length - total_transferred;
// Splice dari block file storage engine ke kernel pipe buffer
ssize_t bytes_in = splice(storage_fd, offset, pipe_fds[1], NULL,
to_transfer, SPLICE_F_MOVE | SPLICE_F_MORE);
if (bytes_in <= 0) {
if (bytes_in == 0 || errno != EAGAIN) break;
continue;
}
// Splice dari kernel pipe langsung ke TCP socket write buffer
ssize_t bytes_out = splice(pipe_fds[0], NULL, client_socket_fd, NULL,
bytes_in, SPLICE_F_MOVE | SPLICE_F_MORE);
if (bytes_out <= 0) {
if (bytes_out == 0 || errno != EAGAIN) break;
continue;
}
total_transferred += bytes_out;
}
close(pipe_fds[0]);
close(pipe_fds[1]);
return total_transferred;
}