Mars optimizes memory use and model choice in long-term AI assistants
MARS: Memory-Aware Routing for Session-Level LLM Assistants
Computation and Language
Summary
When AI assistants chat for a long time, they need to decide what past information to remember and which model should answer each question. The authors present MARS, a method that links memory management and model choice using a shared session state, so the assistant can balance answer quality, memory use, and computation cost. Tests show MARS keeps almost the same answer quality as using full memory but with much less cost, making it more efficient for long conversations. This approach helps AI assistants remember important details while staying fast and cost-effective.
What this means in practice
- •For conversational ai developers: Build long-term AI assistants that efficiently manage memory and choose models for cost-effective, high-quality responses in extended chat sessions.
- •For customer support platform teams: Improve chatbots by reducing API costs while maintaining answer quality through optimized memory views and model selection during multi-turn conversations.
Authors
Minghan Wang, Jinhang Zuo, Zhongxiang Dai, Zhiyong Wang, Fang Kong
Abstract
Long-term LLM assistants must repeatedly decide what prior context to expose and which model should answer as an interaction evolves. Existing memory systems and model routers usually make these decisions separately, although memory construction changes the evidence, distractors, and difficulty of the resulting input. We introduce MARS, a memory-aware routing framework that uses session state as a shared control interface between memory construction and model selection. A Query and Topic Manager (QTM) estimates compact decision-relevant state that guides a fixed state-conditioned memory policy and an online router, enabling each turn to receive an appropriate memory view and a suitable model under a quality-memory-cost trade-off. We evaluate MARS on RealMem and MultiChallenge with 16 system configurations each, covering long-history memory-driven interaction and bounded sessions with structured context interference. On RealMem, MARS preserves 98.2% of full-context GPT-4o's QA while using only 14.5% of its generation API cost, including both generation and controller calls for MARS, and achieves the strongest measured memory-recall and retrieval scores among compared systems. On MultiChallenge, MARS trails the strongest static system overall but approaches its performance at lower recorded cost on a mechanism-focused subset. Together, these results support a favorable quality-memory-cost trade-off for long-history, memory-dependent interaction.