Waiting at the front door: Continuous monitoring of latency in the host network stack
2026-06-01 • Networking and Internet Architecture
Networking and Internet Architecture
AI summaryⓘ
The authors explain that when internet data moves very quickly, delays inside the computer itself can slow things down. They made a tool called netstacklat that watches where delays happen inside the computer's networking system, from the beginning in the Linux network stack to when the app receives data. They tested their tool with different web server setups and found it adds only a tiny bit of extra delay compared to earlier tools. They also used netstacklat in a real-world network to better understand these delays.
latencynetwork stackLinux kernelCPU network interconnecthost networkingHTTP workloadsNginxApachemonitoring tooltail latency
Authors
Simon Sundberg, Anna Brunstrom, Simone Ferlin-Reiter, Jesper Dangaard Brouer, Toke Høiland-Jørgensen
Abstract
With networking moving into the sub-millisecond latency domain, latency in the end host itself can become a significant barrier to achieving consistently low application latency. Both the physical interconnect between the network card and the CPU, the kernel network stack, and the scheduling of applications themselves can be considerable sources of latency. Previous work has studied host latency at various levels, yet there remains a lack of methods and tools to continuously monitor host latency in production. To remedy this, we present netstacklat, a monitoring tool that captures latency at several points in the host network, from the early parts of the Linux kernel network stack all the way until the application reads the data. We evaluate netstacklat in a testbed, demonstrating its ability to capture host latency across 144 variations of HTTP workloads for Nginx and Apache, while also showing how the low monitoring overhead does not inflate tail latency by more than 6%, where previous monitoring solutions increase it by over 100%. Furthermore, we share our initial findings from deploying netstacklat in Cloudflare's global CDN network.