Benchmarking Indonesian Cloud Infrastructure: Network Peering, IOPS Realities, and Kernel Tuning

Deploying production infrastructure in Indonesia presents distinct architectural trade-offs. While hosting compute instances in Jakarta (JKT) yields round-trip times (RTT) under 10ms for domestic users, navigating the local hosting ecosystem requires a solid understanding of hypervisor oversubscription, unpredictable transit paths, and storage fabric bottlenecks. Many engineering teams migrate workloads from Singapore (AWS ap-southeast-1 or GCP asia-southeast1) to local Indonesian providers—such as Biznet Gio, IDCloudHost, Dewaweb, or CBN Cloud—aiming to satisfy data residency laws (PP PSTE) or trim egress latency. However, they frequently encounter volatile disk I/O, CPU steal spikes, and erratic packet routing across fragmented domestic internet exchanges. Evaluating local providers requires looking beyond the marketing copy of 'unmetered bandwidth' and 'enterprise NVMe storage.' You must inspect the low-level hypervisor configuration, audit BGP routing topologies across domestic exchanges, and run real I/O and network stress tests.

The Peering Topology: IIX, OpenIXP, and Eyeball Routing

Indonesia has a unique networking landscape. Domestic traffic relies heavily on local Internet Exchange Points (IXPs): IIX (Indonesia Internet Exchange, operated by APJII), OpenIXP (operated by IDC Indonesia), and newer alternatives such as CDIX. Ideally, traffic between two local endpoints traverses an IXP over direct Layer 2 peering without exiting the country. In reality, asymmetric routing and restrictive transit peering complicate this setup. PT Telkom (the incumbent carrier operating IndiHome and Telkomsel) famously enforces asymmetric peering policies. Telkom does not peer freely at open IXPs with every domestic hoster; instead, traffic often routes through domestic transit links or, in the worst cases, hairpins internationally through Singapore before returning to Indonesia.

TEXT

[Client: Telkomsel LTE] 
       |
       v (International Peering / Transit)
[Equinix SG1 / Singapore]
       |
       v (Upstream Transit Link)
[JKT Transit Provider]
       |
       v
[Local Cloud Hypervisor (Jakarta Cyber 1 / Duren Tiga)]

When testing a domestic provider, do not rely on simple ICMP ping requests. Trace path symmetricity, verify Autonomous System Numbers (ASNs), and identify border routers using Layer 4 probes:

BASH

# Run modern trace routing with ASN and TCP SYN probes to port 443
traceroute -T -p 443 -A target-node.co.id

# Or use mtr over TCP with detailed stats
mtr --tcp --port 443 --report-wide --report-cycles 100 target-node.co.id

If you see high RTT (>30ms) between two nodes located within the Greater Jakarta Area (Jabodetabek), packets are traversing international transits or congested domestic backbones. Ensure your provider peers directly with Telkom Eyeball (AS17974/AS7713), Indosat Ooredoo (AS4761), and XL Axiata (AS24203), alongside having saturated ports on IIX and OpenIXP.

Hypervisor Contention: CPU Steal and Memory Arenas

Budget and mid-tier Indonesian hosting providers heavily utilize KVM, Proxmox VE, or OpenNebula clusters. To maximize margins, providers often overcommit physical CPU cores at ratios exceeding 4:1 or even 8:1. In Linux virtual machines, CPU steal time (%st in top or mpstat) occurs when the guest OS wants to execute instructions, but the hypervisor scheduler (e.g., Linux CFS on the host) is servicing other virtual machines sharing the same physical core. High steal time degrades predictable latency profiles, leading to tail latency spikes in API services and degraded database transactions. Execute an automated load audit to measure noisy-neighbor interference on your virtual machine:

BASH

# Monitor CPU steal metrics over 60 seconds with 1-second intervals
mpstat 1 60 | awk 'NR>3 {print strftime("%Y-%m-%d %H:%M:%S"), $NF}'

# Stress 2 vCPUs while observing context switching and steal rate
stress-ng --cpu 2 --timeout 30s --metrics-brief

If %st consistently climbs above 3% during continuous off-peak and peak workloads, the hypervisor core is severely oversubscribed. For mission-critical workloads like PostgreSQL, Redis, or Kafka, choose providers offering dedicated cores (1:1 vCPU-to-pCPU mapping), or negotiate bare-metal dedicated hypervisors.

Storage Architecture: The Ceph vs. Direct-Attached NVMe Reality

Storage performance is another area where domestic hosting providers show high variance. A provider might claim 'All-NVMe SSD' infrastructure, but the implementation often relies on a software-defined storage pool (e.g., Ceph RBD) backed by a congested 10GbE network fabric. When hundreds of guest VMs issue random I/O operations simultaneously, the network backplane experiences TCP buffer exhaustion and queue congestion. This leads to latency spikes in write acknowledgments that lock the Linux kernel jbd2 thread on ext4 or xfs filesystems. Benchmark pure block performance directly against the raw partition using fio to bypass host page caching:

BASH

# Test 4k Random Write IOPS and tail latency (fsync bound)
fio --name=randwrite --ioengine=libaio --iodepth=16 --rw=randwrite \
    --bs=4k --direct=1 --size=2G --numjobs=2 --runtime=60 \
    --group_reporting --filename=/var/tmp/fio_test_file

# Test sequential read throughput
fio --name=seqread --ioengine=libaio --iodepth=32 --rw=read \
    --bs=128k --direct=1 --size=4G --numjobs=1 --runtime=60 \
    --group_reporting --filename=/var/tmp/fio_test_file

Evaluate the clat (completion latency) percentiles carefully. A solid cloud provider should exhibit a 99.9th percentile (p99.9) under 5 milliseconds for 4K random writes. If your p99.9 exceeds 80ms, transaction commits on relational databases will lock worker pools during traffic bursts.

Architectural Benchmark Comparison

Below is an engineering comparison of typical infrastructure tiers available in the Indonesian market:

Feature / Metric	Tier 1: Indonesian Enterprise Cloud (e.g., Biznet Gio, CBN Cloud)	Tier 2: Developer-Focused VPS (e.g., IDCloudHost, Dewaweb, Kilat Storage)	Tier 3: Budget cPanel/Shared VPS Hosters	Global Public Cloud in JKT Region (AWS, GCP, Alibaba Cloud)
Underlying Hypervisor	VMware ESXi / Nutanix / Enterprise KVM	Open-source KVM / Proxmox VE	Virtuozzo / Overcommitted KVM	Nitro / KVM heavily customized
vCPU Overcommit Ratio	Typically 1:1 or 2:1	3:1 to 6:1	8:1 or higher	Guaranteed vCPU (unless burstable 'T' instances)
Storage Fabric	SAN (Fibre Channel) or 40GbE Ceph NVMe	10GbE Ceph / Local ZFS / Direct NVMe	Local SATA/SAS SSD Arrays	Distributed EBS / Persistent Disk Pools
Domestic Peering (IIX/OIXP)	10Gbps - 40Gbps redundant ports	1Gbps - 10Gbps shared ports	1Gbps shared	Direct private interconnect + Tier-1 transits
Telkom Eyeball Pathing	Optimized direct domestic peering	Variable (sometimes via transit)	Often throttled / Asymmetric	Direct domestic bilateral interconnects
Observed 4K Random Write IOPS	15,000 - 45,000 IOPS	2,000 - 12,000 IOPS	500 - 2,500 IOPS	3,000 - 64,000+ IOPS (provisioned)
Average Domestic RTT (Jabodetabek)	2ms - 8ms	3ms - 15ms	5ms - 35ms	2ms - 6ms
Compliance (PP PSTE / OJK)	High (Certified, audited DC)	Medium (Varies by provider facility)	Low	High (SOC2, ISO 27001, OJK aligned)

Linux Kernel Tuning for Domestic Environments

To maximize performance on Indonesian VPS instances and mitigate network jitter, tune the Linux kernel network stack and virtual memory subsystems. Apply these optimizations via /etc/sysctl.d/99-network-performance.conf:

INI

# Increase TCP receive and transmit buffer maximums
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Use BBR congestion control to prevent bufferbloat across local ISP hops
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Adjust queue lengths for high-concurrency domestic traffic
net.core.somaxconn = 8192
net.core.netdev_max_backlog = 16384

# Mitigate sudden I/O pauses on overcommitted virtualization storage
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
vm.swappiness = 10

Apply the configurations using sysctl -p /etc/sysctl.d/99-network-performance.conf. Switching the TCP congestion control algorithm to bbr (Bottleneck Bandwidth and RTT) is particularly beneficial in Indonesia. Local fiber routes can experience intermittent packet drops due to physical link fluctuations or congested regional exchange links. Unlike Cubic, which drops its congestion window by half upon detecting packet loss, BBR models the network pipe based on actual delivery rate and round-trip propagation time, maintaining stable throughput. Verify that the kernel loaded BBR successfully:

BASH

sysctl net.ipv4.tcp_congestion_control
# Output should be: net.ipv4.tcp_congestion_control = bbr

Mitigating High-Risk Failure Modes

Operating in the Indonesian datacenter ecosystem requires accounting for several localized operational risks:

Gedung Cyber Outages: A significant portion of local exchange switching infrastructure has historically been concentrated in Gedung Cyber 1 (Kuningan, Jakarta). While redundant facilities (such as Cyber 2, IDC Duren Tiga, and Cikarang tech parks) have emerged, ensure your provider does not depend on a single physical cross-connect in older downtown facilities that have historical vulnerabilities to electrical and cooling disruptions.
Submarine Cable Breaks: The Bangka-Batam-Singapore (B2JS) and Jakarta-Bangka-Batam-Singapore (JAKABARE) cable routes run through active maritime corridors in the Java and Malacca Seas. Anchor drops and dredging frequently cause fiber cuts. If your architecture relies on external APIs (e.g., Stripe, AWS S3, or third-party identity providers in Singapore), design asynchronous message queues (e.g., RabbitMQ or Kafka) with local fallback storage to keep services resilient during regional transit outages.
Local MTU Mismatches: Certain domestic cloud providers encapsulate traffic within VXLAN or GRE overlays without enabling Jumbo Frames on physical top-of-rack switches. This configuration lowers the effective Maximum Transmission Unit (MTU) below the standard 1500 bytes. If Path MTU Discovery (PMTUD) is blocked by aggressive ICMP drop rules at the host firewall, TCP packets exceeding 1450 bytes can hang on TLS handshakes.

Audit your interface MTU path via:

BASH

# Test maximum non-fragmented frame size to your gateway
ping -M do -s 1472 local-gateway-ip

If packets drop, reduce your network interface MTU to 1450 or 1400 inside your network configuration (netplan or /etc/network/interfaces) to prevent silent TCP stall conditions.

Engineering Assessment Checklist

Before signing a contract or committing your production workloads to an Indonesian hosting provider, run this technical checklist across a staging instance:

CPU Provisioning: Run stress-ng for 15 minutes; confirm that %st in mpstat remains below 1%.
I/O Latency Under Load: Measure fio 4K random write latency (p99.9 under 10ms).
Peering Symmetry: Verify outbound and inbound routes through Telkom (AS17974) and Indosat (AS4761) without hops transiting Singapore.
MTU Validation: Confirm PMTUD works as expected across the virtual network interface.
Facility Redundancy: Confirm the datacenter deployment tier (minimum Tier III design topology with N+1 or 2N electrical pathing outside single-point legacy buildings).

Architecting workloads within Indonesia provides clear latency benefits for domestic users, but it requires disciplined performance verification. Running thorough benchmarks against the hypervisor, storage layer, and domestic routing paths ensures your application maintains reliability when scaling under real-world local traffic.