Multimodal memory method improves long-term AI agent reasoning
MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents
Computation and Language
Summary
Large language model (LLM) agents need to remember important information over many steps, but their memory space is limited. The authors created a system called MEMO that chooses what to remember and whether to store it as text, images, or both, depending on what helps later tasks the most. MEMO organizes memory efficiently so the AI can find and use key facts better, improving its performance on several question-answering and task benchmarks. This method balances detail with space, making long-term reasoning more effective.
What this means in practice
- •For chatbot developers: Create chatbots that remember and organize past conversations efficiently to improve responses over long interactions.
- •For interactive ai game designers: Build AI characters that recall and use prior game events better in complex, multi-step task environments.
Authors
Xian Gao, Jinpeng Wang, Jiacheng Ruan, Guangyu Cao, Ting Liu, Yuzhuo Fu
Abstract
Long-running LLM agents rely on external memory to store and reuse information beyond a single context window, yet there is a fundamental tension between the continuous accumulation of interaction trajectories and the limited context capacity. The key challenge in agent memory is therefore not only to retrieve relevant records, but also to select necessary evidence under a given budget and organize it in an appropriate modality. Existing memory readout methods mainly use textual or visual forms. Text preserves high fidelity, but its linear token representation makes contents with different importance compete for the limited context at nearly uniform unit cost. Visual readout renders text into document-like images, which can use two-dimensional layouts to expose structure and emphasize key information, but it may lose fine-grained details during rendering and compression. To address this issue, we propose MEMO, a multimodal evidence memory organization method for LLM agents. MEMO first uses a trained evidence extractor to select relevant memory blocks and form evidence units with source information and presentation requirements. A trained query-conditioned memory manager assigns each unit to a textual, visual, or dual-channel carrier and selects a layout that matches the evidence structure. A deterministic memory construction module then generates the textual package and visual pages. The memory manager is trained with feedback from an offline reader that measures the utility of the guided memory plan, so that retention and presentation decisions align with downstream usage. We evaluate MEMO on four benchmarks, HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld, with multiple reader backends. The results show that MEMO presents memory more efficiently with fewer memory tokens, improves downstream task performance, and builds more effective working memory under constrained budgets.