TideCache improves large language model speed and memory use on CPU GPU chips

TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

Distributed, Parallel, and Cluster ComputingMachine Learning

Summary

Large language models need different types of memory cache to work efficiently, especially when handling long or many conversations. The authors present TideCache, a system that smartly manages memory between fast GPU and slower host memory to speed up these models. TideCache uses three parts to share memory, prefetch data while computing, and decide how to run requests to save space and time. Tests show TideCache can handle more tasks faster and with less waiting compared to previous methods.

What this means in practice

  • For llm infrastructure teams: Run large language models more efficiently by better managing memory caches to reduce latency and increase throughput on CPU GPU superchips.
  • For cloud service providers: Increase the number of large-language-model requests handled simultaneously with lower delay by using smart caching and streaming techniques.$Commercial implications: This enables cloud vendors to offer faster, more scalable AI services that better utilize their hardware.

Authors

Jinbin Luo, Xiaozhe Yao, Shang Yang, Zhijian Liu, Song Han, Qinghao Hu

Abstract

Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.