Pull reduces token costs in long LLM conversations with lazy loading
Pull: Lazy Materialization of Working Memory for Stateful LLM Conversations
Computation and Language
Summary
Long conversations with language models get slower and more expensive because the model has to remember everything, which grows quickly with the number of turns. The authors created Pull, a method that stores conversation history in a smart way so the model only loads parts it needs at each step. This keeps the conversation quality the same while greatly reducing the amount of context the model uses. Pull can also reopen past conversation parts if needed, unlike other methods that lose some details permanently.
What this means in practice
- •For chatbot developers: Cut down the computational cost of maintaining long chat histories without losing important context for better user conversation continuity.
- •For customer support platform teams: Keep longer interactions accessible on demand while reducing the processing needed for real-time assistance with AI chat companions.
Authors
Jiangang Chen
Abstract
As LLM conversations grow to hundreds of turns, full-context injection incurs $O(N^2)$ cumulative token costs, while lossy summarization or hard truncation irreversibly discards historical state. We propose Pull, a session router that maintains an addressable metadata directory via a local, deterministic Purifier (zero LLM calls, millisecond-level latency). At query time, the LLM lazily materializes only the turns it needs; unmaterialized turns remain accessible but collapsed. Unlike irreversible compression, Pull's materialization is reversible; subsequent queries can expand any collapsed turn. On LoCoEval (128 conversations, 12,780 turns), Pull reduces per-query context tokens (Phase 2) by 75.1 percent on single-hop tasks with equivalent quality ($Δ= -0.002$, n.s.) and by 72.0 percent on multi-hop tasks with no quality loss ($Δ= +0.017$). A controlled routing benchmark (7,831 queries x 10 methods) shows that entity lifecycle tracking is empirically a prerequisite for distance-independent routing. On BEAM 1M (14 conversations, 263 questions), Pull improves F1 by +55.2 percent over a truncation baseline.