Chiron uses llm advice to improve cache speed for ai workloads
Chiron: Cache Management Empowered by LLM Intelligence for AI Workloads
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Many AI programs need to quickly access data stored far away, so they use local caches to save time. But picking the right way to manage these caches is tricky because different AI tasks use data differently and change over time. The authors introduce Chiron, which uses large language models (LLMs) to guess what kind of data access will happen next and suggest how to manage the cache. Instead of relying only on the LLM's advice, Chiron checks and applies expert rules to keep things fast and efficient. Their tests show Chiron can speed up AI tasks by a noticeable amount compared to older methods.
What this means in practice
- •For ai infrastructure teams: Improve AI cluster performance by adapting cache policies dynamically based on detected access patterns without changing user code.
- •For cloud storage operators: Deploy advanced cache management that uses LLM insights to reduce data waiting times and boost system throughput for AI workloads.
Authors
Bohan Zhang, Renshu Ge, Tianze Wang, Chen Chen
Abstract
AI clusters keep data in remote object stores and rely on local caches to stop GPUs from waiting for data. A cache performs well only when its prefetching and eviction policies match the access pattern, yet these patterns differ across AI workloads and change between phases of one workflow. Existing methods either need code changes, which compromises user transparency, or use low-end heuristics, which cannot follow such changes. Large language models (LLMs) can infer access behavior without user hints, but direct LLM control of cache actions suffers from long reasoning delay, high token cost, and poor decision quality. We present Chiron, a cache system based on advisory switching control. An LLM asynchronously diagnoses the access pattern and recommends a policy. A deterministic harness gates this advice, executes expert-designed policies locally, and returns cache feedback for review. We implement Chiron atop JuiceFS and evaluate it on nine AI workloads. It reduces job completion time by 14.3-41.2% over the best baseline on popular AI workloads.