Personal agents improve knowledge use with memory guided retrieval
ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents
Artificial IntelligenceComputation and Language
Summary
Personal assistant programs often need to remember past conversations to give better answers based on what users need now. The authors created ColaBench, a tool to test how well these assistants use past information and new facts together, especially in health and daily life topics. They also built MemKB, a system that uses memory to find and apply knowledge correctly, scoring about 86% accuracy in tests. This work helps improve how digital helpers understand and use what they've learned about you over time.
What this means in practice
- •For personal assistant developers: Create more accurate assistants that use users’ interaction history to retrieve and apply relevant knowledge for personalized responses.
- •For healthcare app engineers: Integrate memory-linked knowledge retrieval into health-focused apps to tailor advice based on users’ past conditions and interactions.
Authors
Wenjie Zhang, Zhichao Shi, Xuhui Jiang, Xiaojun Wu, Cehao Yang, Chengjin Xu, Xiaoning Song
Abstract
Personal agents need to draw on long-term interaction history to retrieve and apply external knowledge suited to a user's current request. Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances. This creates two connected challenges: using historical conditions to identify the knowledge needed for a new request, and applying that knowledge to the user's situation. Existing benchmarks advance memory recall, personalized retrieval, and memory-guided action, while the joint evaluation of these two challenges remains underexplored. We introduce ColaBench, a synthetic benchmark in health and daily living, with 349 tasks across 10 Profiles and a knowledge base of 13,033 documents. Its Evidence Graph connects personal conditions to their disclosures across public interaction sessions, records prior document exposure, and links answer requirements to personal and newly acquired document evidence. We also introduce MemKB, which uses state summaries to inform queries and rule application, and memory-document links to retrieve knowledge associated with the user's history. MemKB achieves 86.25% Accuracy with an average of 4.07 interaction rounds per task. Task-level analysis finds larger gains on tasks integrating information across multiple historical Facts; error analysis further identifies unsupported personal assumptions and rule misapplication as obstacles to using retrieved evidence. ColaBench provides traceable tasks and an evaluation framework for studying how memory guides knowledge acquisition and rule application.