Predicting performance of AI model clusters with hierarchical caching strategies
Compositional Performance Modeling of LLM Inference Clusters with Hierarchical KV Caching
PerformanceDistributed, Parallel, and Cluster Computing
Summary
Running big AI language models on clusters of computers involves complex interactions that affect how quickly answers come back. The paper introduces a simulator that predicts how different hardware setups and caching methods impact speed without needing to test each setup in full. The authors tested their simulator with real AI models and found it accurately estimates response times. This helps teams choose the best ways to set up and manage AI clusters to reduce delays and improve efficiency.
What this means in practice
- •For ai infrastructure teams: Estimate latency and resource use for different large language model cluster setups to optimize hardware and caching policies.
- •For cloud platform engineers: Simulate how shared resources and caching strategies affect AI service responsiveness to improve cluster scheduling and policy design.
Authors
Ruiyang Ma, Sheng Qi, DaiKang Kuang, Teng Ma, Guojie Luo
Abstract
Design-space exploration of LLM inference clusters requires predicting how hardware and serving policies interact with hierarchical key-value (KV) caching. Isolated operation profiles alone do not determine cluster latency: reuse changes computational demand, while KV location, capacity, and contention govern execution order. We present a compositional discrete-event simulator based on separating local service costs from shared resource and KV availability. Compute and transfer models calibrated on one-prefill/one-decode (1P-1D) deployments are reused across compatible workers, enabling policy and topology comparisons without per-configuration latency fitting. Validation against SGLang covers Qwen3-8B and Qwen3-32B on 1P-1D and 3P-1D clusters. Mean request end-to-end latency error is 1.34% for question answering and 6.36% for captured coding sessions, measured as mean absolute percentage error. CPU simulation reduces replay turnaround by 15.33 times on average; real replay includes prescribed arrival gaps. Measured session-affinity and simulated retention studies demonstrate how the framework guides policy selection by predicting the joint effects on computation, latency, and KV occupancy.