Linux large reads reveal wait time causes and enable performance tuning
Behind a Simple Read: Understanding Work and Waiting in the Linux I/O Stack
Operating Systems
Summary
Reading data in Linux may seem like a simple operation, but its speed depends on many hidden steps and waiting times. The authors studied how big read requests are handled, showing how the system overlaps data copying with device waiting to save time, but these benefits rely on things like memory layout and cache state. They built a model to predict waiting times accurately and used it to understand when speeding up parts of the system helps or not. Their work helps explain why read performance varies and guides better design and tuning of Linux I/O.
What this means in practice
- •For linux kernel developers: Optimize read performance by predicting and controlling wait times within the Linux I/O stack using the authors’ wait chain and device models.
- •For storage system engineers: Design storage backends and scheduling policies that better overlap data copy and device wait to improve throughput and latency in SSD-based systems.
Authors
Yang Shen, Kai Lu, Min Xie, Huijun Wu, Zhenwei Wu, Wenzhe Zhang
Abstract
A simple read interface provides uniform functional semantics, but not equally simple or predictable performance behavior. We use synchronous large-buffer reads in Linux as an observation window and decompose the buffered-read path end to end, distinguishing work volume, processing time, and critical-path exposure. We find that Linux reduces metadata work through large folios and overlaps most cold-read copying with device waiting, but these mechanisms depend on access advice, folio granularity, cache state, and backend execution. Using an experimental kernel prototype, we validate additional opportunities from out-of-order early copying and opportunistic parallelism, while showing that faster request submission does not improve end-to-end performance when SSD supply is already sufficient. To explain device waiting, we abstract first-completion wait $F$ and subsequent completion capacity $B_{CQ}$ from finite request-batch completion timelines. Measurements on a real SSD characterize how they vary with request size and batch size, while MQSim experiments connect them to internal device mechanisms and distinguish finite-batch completion from sustained throughput. We then construct a cross-layer wait chain along the actual folio--bio--request mapping. Independently calibrated device parameters predict waiting at the block layer and at read_pages with errors no greater than approximately 6.9% and 3.2%, respectively. Finally, four use cases apply the model and wait chain to parallel submission, Linux readahead, dependent-read layout, and polling versus sleeping. Their gains, no gains, and gain reversals form a closed loop from observation and modeling to explanation and control, providing a measurable basis for performance decisions in layered I/O stacks.