MapFlow speeds up large language model use by reusing attention patterns
Accelerating Cross-Model LLM Inference via Attention Map Reuse
Machine Learning
Summary
Running multiple different large language models often involves re-processing the same background information repeatedly, which can be slow. The authors present MapFlow, a system that speeds this up by reusing attention maps—patterns showing how words relate to each other—that can transfer between different models. Instead of saving detailed model-specific states, MapFlow stores and adapts these attention patterns, reducing wait times while keeping the output quality. They also use clever compression and retrieval techniques to handle very long inputs efficiently.
What this means in practice
- •For ai service operators: Speed up serving pipelines that run multiple different large language models on shared input data by reusing attention maps across models.
- •For business automation teams: Improve efficiency of multi-step language-model-driven workflows that involve diverse tools and interaction histories by reducing redundant computations.
Authors
Zhan Jin, Yuze Luo, Bin Cui, Xupeng Miao
Abstract
LLM-augmented data workflows often involve multiple heterogeneous models operating over substantial shared context, such as schemas, tool specifications, interaction history, and intermediate results. Although prefix caching can reduce prefilling overhead, existing approaches are largely limited to reuse within the same model because KV states are tightly coupled to model-specific architectures. We present MapFlow, a system for efficient heterogeneous LLM serving via cross-model attention map reuse. Instead of reusing KV states, MapFlow reuses attention maps, which capture token dependency patterns and are more transferable across models. To enable efficient and accurate reuse, we combine partial recomputation with lightweight linear translation across layers and heads. To make attention reuse practical for long contexts, we further introduce block-level sparse compression and a fused retrieval kernel. Experiments show that MapFlow reduces prefilling latency and improves end-to-end serving efficiency while preserving generation quality across heterogeneous model pairs and multi-turn routing workloads. These results suggest that attention maps provide an effective reusable state for heterogeneous LLM serving, especially in prefix-heavy data workflows. Our code is available: https://github.com/PKU-DAIR/MapFlow.