Efficient kv cache sharing method speeds up multi-lora agent inference
PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction
Machine Learning
Summary
When AI agents share a common large model but have different roles, they each keep their own memory of past steps, which wastes time and memory. The authors propose PReCache, a way to share those memories more efficiently without extra training. PReCache uses a shared base memory with small extra parts customized for each agent, letting each agent skip repeating shared work. Their approach speeds up processing and keeps accuracy close to normal methods.
What this means in practice
- •For ai inference engineers: Speed up role-specialized multi-agent inference by efficient KV cache sharing without retraining or architecture changes.
- •For cloud ai service providers: Increase throughput and reduce latency in serving multiple specialized agents by sharing base computation in KV caches.
Authors
Hyesung Jeon, Hyeongju Ha, Jae-Joon Kim
Abstract
Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the growing shared trajectory and constructs its own KV cache, introducing substantial memory and computation redundancy in long-horizon tasks. Existing KV cache sharing methods reduce this repeated prefill, but they either require additional training or architectural constraints or retain substantial model computation. Moreover, direct cache reuse causes the current agent to rely on cache states generated by the previous agent's adapter, weakening the role-specific behavior encoded by its own LoRA. We present PReCache, a training-free KV cache sharing framework with two designs, namely PreLRShared and ReBaseShared, that share the base cache computed using the pretrained weights and precompute a compact agent-specific low-rank (LR) cache. To remove repeated prefill, PreLRShared precomputes each agent's LR cache when the shared context is first processed, allowing the current agent to use its own LR cache without reprocessing context processed by previous agents. To improve sharing accuracy, ReBaseShared reconstructs the shared base cache from adapter-free hidden states, reducing the remaining error caused by the previous agent's adapted representation. To minimize its reconstruction cost, we propose two inference schemes tailored to single-stream inference and concurrent serving, performing the same reconstruction after each agent's turn or alongside its execution, respectively. Across multiple models and agent benchmarks, PreLRShared achieves up to a 3.1x TTFT speedup and a 2.3x improvement in per-request throughput over inference without KV cache sharing. ReBaseShared best preserves accuracy overall among the evaluated cache-sharing methods, with an average drop of only 1.1 points relative to inference without cache sharing.