Kernel I/O flushing limits cloud database speed and latency
Invisible Yet Dominant: Big Stalls of Kernel I/O Mechanisms in Cloud OLTP Databases
DatabasesOperating Systems
Summary
Databases like PostgreSQL depend on the Linux kernel to manage writing data to disk, but the authors found a hidden slowdown in how the kernel handles this in cloud environments. A single thread per device handles flushing data, which can cause delays invisible to standard monitoring tools. The authors used special kernel probes to measure this stall and discovered that adding more of these flushing threads, rather than just increasing bandwidth, significantly cuts delays and speeds up database transactions. This finding could help make cloud databases faster and more reliable.
What this means in practice
- •For cloud database engineers: Optimize cloud database configurations by using multiple kernel write-back threads to reduce transaction latency and improve throughput under load.
- •For cloud storage architects: Design distributed block storage that supports multiple flush drains to avoid invisible stalls and maximize performance in cloud database workloads.
Authors
Mitsumasa Kondo
Abstract
Most databases, including PostgreSQL, RocksDB, and recent AI KV-cache middleware, rely on buffered I/O, delegating write-back to the Linux kernel. On the distributed block storage standard in the cloud, this delegation inherits a hidden bottleneck: each device is drained by a single kernel flusher thread over a high-latency, shallow-queue path. When the drain falls behind, dirty throttling pauses write() system calls, and even reads that must evict dirty pages stall. These stalls are invisible to iostat and every standard counter. This poster observes the stall from inside the kernel, using the multi-volume data placement proposed in SteelDB as the experimental lever. eBPF probes on writeback and block tracepoints separate write-back by issuing context and count every throttle pause. Across three configurations with identical provisioned IOPS and bandwidth but 1, 2, and 4 devices, we show that adding drains, not bandwidth, cuts throttle pauses by 70%, reduces maximum transaction latency by 59%, and raises throughput by 23%.