SparseEngine boosts large language model speed and memory efficiency
SparseEngine: Sparse-First Inference Engine
Machine Learning
Summary
Handling long conversations with large language models (LLMs) uses lots of memory and slows down processing. The authors created SparseEngine, a new tool that organizes and manages memory more efficiently by focusing on sparse attention methods. It supports many ways to store and prune information, making LLM use faster by over twice or even ten times in some cases. This helps keep the quality of responses while speeding up real-world applications like chat agents.
What this means in practice
- •For large language model engineers: Run long-conversation LLM applications more efficiently by incorporating SparseEngine’s memory and attention management to reduce latency and computational load.
- •For chatbot platform developers: Integrate SparseEngine to speed up multi-turn dialogue processing in chatbots while reducing memory costs during user interactions.
Authors
Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu
Abstract
Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.