New benchmark measures agent memory with cost and speed tradeoffs
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Computation and LanguageArtificial Intelligence
Summary
Many AI agents need to remember information from past interactions to do real-world tasks well, but current tests only check if they get answers right. The authors created DolphinBench, a new test that measures how well AI agents use memory to finish jobs, including considering how fast and cheaply they do it. They made scenarios with lots of past messages and tasks that rely on this history, testing agents by seeing if they succeed only when using the right memory. They also ask developers to report time and cost along with accuracy to better understand memory system tradeoffs.
What this means in practice
- •For software engineers: Evaluate AI assistant memory strategies by measuring task success, speed, and cost using realistic user histories provided by DolphinBench.
- •For chatbot developers: Improve chatbot memory systems by testing them under scenarios that demand context recall with strict cost and latency constraints.
Authors
Soumil Rathi, Deshraj Yadav, Taranjeet Singh
Abstract
Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.