High iowait in a Proxmox VM: Is It the Guest Disk or the Host Storage?

Your VM on Proxmox feels sluggish. SSH takes a moment to respond, and top shows a high number in the wa column. The obvious conclusion is that the VM's disk is overloaded, but inside a VM that number can't tell you which disk. It might be your own workload hammering the virtual disk, or it might be another VM on the same host storage, and both look the same from where you are sitting. This guide walks through checking both layers with standard CLI tools, in the same time window, so you can say which one is saturated instead of guessing. It is also the question a support engineer usually has to answer first, because the fix is completely different depending on the answer.

What iowait actually measures

iowait is CPU time spent idle while at least one I/O request from that CPU is still outstanding. It is a CPU accounting state, not a disk metric. A busy CPU with heavy I/O can show low iowait, because the CPU never sits idle. A mostly idle VM waiting on a single slow request can show high iowait. So the number tells you processes are waiting on storage, but not how hard the storage is working or where the waiting happens. In a KVM guest, every request to the virtual disk passes through QEMU on the host and then down to the real storage: a local ZFS pool, LVM-thin, a RAID array, Ceph, or NFS. A delay anywhere along that path shows up in the guest as iowait. The guest has no visibility past its own virtual device. Check one thing in top before going further: the st (steal) column. High steal means the guest is waiting for physical CPU time on the host. That is a different problem from storage contention, though both can happen on an overcommitted node. If steal is high and wa is modest, look at host CPU first.

Measure the guest side first

Inside the VM, you want two things: how much I/O the guest is issuing, and how long each request takes. iostat from the sysstat package covers both. Install sysstat through your distro's package manager if it is missing. pidstat (same package) shows which process is responsible, and /proc/pressure/io (PSI, pressure stall information) shows how much time tasks spent stalled on I/O, if your kernel supports it.

BASH

# Extended per-device stats every 1 second; ignore the first report (it's an average since boot)
iostat -x 1

# Per-process read/write rates every 1 second
pidstat -d 1

# I/O pressure: share of time tasks were stalled on I/O over 10s, 60s, 300s windows
cat /proc/pressure/io

In the iostat output, focus on the virtual disk (usually vda for VirtIO Block or sda for VirtIO SCSI) and these columns:

r/s and w/s: requests per second. This is how much work the guest is asking for.
rkB/s and wkB/s: throughput.
r_await and w_await: average milliseconds per request, including time spent queued. This is the latency the guest actually experiences.
aqu-sz: average number of requests in flight.

Skip %util inside a VM. It only measures the share of time the device had at least one request in flight. A virtual disk can serve many requests in parallel, so 100% there does not mean the device is full. Await plus request rate is far more useful. Older sysstat releases name some of these columns differently (await, avgqu-sz), but the meaning is the same. What you are looking for is the relationship between volume and latency. If await is high and the guest is also pushing a lot of requests, with pidstat showing a database or backup job writing heavily, the guest is very likely the source. If await is high while the guest is issuing only a trickle of I/O, the delay is coming from below. A small number of requests should not take that long on healthy storage. As an illustrative example: a guest showing around 20 writes per second with w_await in the hundreds of milliseconds is not generating meaningful load on its own. Something underneath is slow.

Check the host while the problem is happening

Timing matters here. Host and guest numbers are only comparable if you capture them in the same window, so run the host commands while the guest is still showing high iowait. Open both sessions side by side if you can. On the Proxmox node, look at the physical devices and at which VMs are generating I/O:

BASH

# Physical device stats (sdX, nvmeXn1, etc.) every 1 second
iostat -x 1

# If the VM disks live on ZFS: per-vdev ops and bandwidth, plus latency columns with -l
zpool iostat -v 1
zpool iostat -l 1

# Host-wide I/O pressure
cat /proc/pressure/io

# Per-process disk I/O for KVM guest processes only
pidstat -d -C kvm 1

# Map the kvm PIDs above to VMIDs (PID column)
qm list

Each running VM on Proxmox is a kvm process, so pidstat -d -C kvm 1 gives you a rough per-VM view of read and write rates. Match the PIDs against the qm list output and you know which VMID is responsible. Some I/O can be attributed to helper threads depending on the disk backend and cache settings, so treat it as a ranking of who is busiest, not exact accounting. On the physical devices, the same logic applies as in the guest: look at request rate, throughput, and await. For spinning disks and SATA SSDs, %util close to 100% together with rising await is a reasonable saturation signal. For NVMe it is weaker, for the same parallelism reason as virtual disks. On ZFS, zpool iostat -l breaks latency into wait and disk components. Long queue waits with modest disk latency usually mean the pool is getting more requests than it can schedule, not that a single drive is failing. If the storage is Ceph or NFS, the node's local disks will not tell you much. The equivalent check moves to the storage cluster or network. Per-process stats for kvm still help identify which VM is busiest.

Reading both sides together

Once you have both sets of numbers from the same window, the pattern is usually clear:

Guest view	Host view	Likely cause
High await, high request rate, one process dominant	Physical disks busy, this VM's `kvm` process at the top of `pidstat`	The guest's own workload is saturating storage
High await, low request rate	Physical disks busy, other VMIDs at the top of `pidstat`	Contention from neighbors on shared storage
High await, low request rate	Physical disks quiet, low host I/O pressure	A limit between guest and storage, often a per-disk throttle
Low await, high `st` in top	Host CPU busy	CPU steal, not storage

The third row is easy to miss. If the host storage is idle but the guest still sees long waits, check the VM's disk options. Proxmox lets you set per-disk bandwidth and IOPS limits, and a VM hitting its cap looks exactly like slow storage from inside. That case has its own walkthrough in How to Check If Proxmox Disk Throttle Limits Are Causing High iowait in a VM. The virtual disk controller can also shape queue behavior under heavy transactional load, which is covered in Perubahan Performa Disk Saat Beralih dari VirtIO Block ke VirtIO SCSI. When the cause is the guest itself, the fix lives in the guest: tune the database, reschedule backups, add caching, or move to a disk tier that matches the workload. When the cause is a noisy neighbor, the guest can't fix it. Someone with host access has to throttle, rebalance, or migrate the heavy VM.

Verifying before and after a change

Whatever you change, measure it the same way you diagnosed it. Capture a baseline first: run iostat -x 1 in the guest and on the host for a minute or two while the problem is visible, and save the output with tee or redirection. After the change, repeat the capture under a similar workload at a similar time of day. Compare await and request rates, not the iowait percentage alone. If you want a controlled test instead of waiting for real traffic, fio can generate a repeatable workload inside the guest. Be careful with it on shared storage. A write-heavy fio run on a production node is exactly the kind of noisy neighbor you are trying to diagnose. Use a dedicated test file, keep runs short, and avoid running it against a raw device that holds data. Also watch for improvements that only move the problem. Lowering one VM's I/O with a throttle can make its own iowait worse while freeing the host for everyone else. That trade-off may be the right call, but it should be a deliberate decision, not a surprise. The practical rule: never diagnose a VM's iowait from inside the VM alone. Pair guest latency and request rate with host device metrics and per-kvm process I/O from the same minute. The mismatch, or the lack of one, tells you which layer is saturated and who has to act on it.