RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts
Artificial Intelligence
Summary
The authors created a test called RECON to see how well large language models can remember and think about very long stories, like criminal or medical case files. These stories are really long, and the test checks if the models can connect clues, handle changes in information, and understand timelines. They found that current models struggle a lot, getting less than a quarter of the answers right. This shows it’s hard for AI to keep track of complicated information over long conversations.
Authors
Mihir Shriniwas Arya
Abstract
Large language models and LLM-based agents are widely used as personal chat assistants, enterprise copilots, and autonomous workflow agents. In all these applications, memory (the ability to retain, access, and reason over information accumulated over long contexts and multiple interactions) plays a crucial role in determining the reliability of any agent. We introduce RECON (Reasoning over Extended Contexts with Obfuscated Narratives), a benchmark for evaluating compositional reasoning over long contexts. RECON spans 24 case files across three domains (criminal, medical, and financial), each ranging from 50k to 100k tokens, and tests agents on six memory intensive tasks: reconstructing multi-hop evidence chains, propagating cascading invalidations, resolving source conflicts, counterfactual reasoning, satisfying temporal constraints, and temporal fact retrieval. Recent memory benchmarks evaluate whether agents can retrieve scattered facts or detect if a fact has changed whereas RECON evaluates what happens after the change, whether agents can trace which downstream conclusions are affected, which survive through independent support, and how alternative timelines would have unfolded. Our evaluation reveals substantial limitations across current architectures: even the strongest non-Oracle system reaches only 22.4% Accuracy, with retrieval and reasoning each surfacing as challenges.