SpecCompact speeds up context compaction for LLM agents by parallel slot verification
SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification
Computation and Language
Summary
Large language model (LLM) agents remember past conversations by summarizing them, but this can be slow and slow down responses. The authors noticed that certain parts of summaries, called slots, can be created independently and checked quickly. SpecCompact is a new method that drafts these slots in parallel with a smaller model, then uses a bigger model to verify and fix any mistakes. This speeds up summarizing histories by about 2 to 5 times in tests, and up to 8 times faster in a user study. The approach involves some trade-offs depending on the task and model used.
What this means in practice
- •For chatbot developers: Speed up conversation history summarization in chatbots by generating and verifying summary parts in parallel to reduce response delay.
- •For virtual assistant engineers: Reduce latency in context compaction within virtual assistants, enabling faster understanding of extended user interactions without losing critical information.
Authors
Yulu An, Yifei Feng, Xiangyang Yin
Abstract
LLM agents use context compaction to manage growing interaction histories, but summarizing long histories with large models can introduce substantial delays. Compaction therefore requires balancing information retention against latency. We observe that structured summary slots can retain information when generated independently. Based on this observation, we present SpecCompact, a framework that accelerates compaction through slot-parallel semantic speculation. SpecCompact fine-tunes a small drafter and generates slots in parallel. The target model verifies each proposed line and repairs rejected content, while subsequent drafting uses the updated slot prefix. System optimizations further reduce repeated context processing and model calls. Across AppWorld and $τ^3$ benchmarks with two target models, SpecCompact achieves $2.04$--$5.63\times$ compaction speedups over Hermes-style summarization. Separate downstream evaluations reveal model- and task-dependent performance trade-offs. A user study within Hermes further shows an $8.68\times$ median compaction speedup.