ProMemAgent improves long-term assistance by remembering future tasks
ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions
Artificial Intelligence
Summary
People often plan to do things much later, like hours or days ahead, but most computer assistants only help with what's needed right now. The authors created ProMemAgent, a system that watches and remembers sensory signals like video and sound to recognize and recall future intentions in the right situations. This system can proactively remind or assist users long after the original event happened. They also built a test collection called CPMBench to measure how well such systems work over many hours or days.
What this means in practice
- •For smart home developers: Create home assistants that remember and act on user intentions hours or days later using sensory inputs like video and audio.$Commercial implications: Enables sale of smart home assistants with advanced long-term reminder features for consumers, improving usefulness beyond immediate commands.
- •For personal robotics teams: Build robots that monitor environment over time and proactively help users with delayed tasks by recalling relevant past context.
Authors
Lilin Xu, Linyang He, Kaiyuan Hou, Yuang Fan, Minghui Zhao, Xuhai Xu, Xiaofan Jiang
Abstract
Large language model (LLM) agents are moving beyond reactive responses toward proactive assistance that anticipates user needs without explicit requests. However, existing proactive agents primarily address immediate needs, while many everyday situations involve delayed intentions that can only be acted on hours or days later. Humans handle such intentions through prospective memory, the ability to remember to act on an intention when the right moment arises. In this paper, we introduce ProMemAgent, the first proactive agent to incorporate prospective memory over everyday sensory contexts for long-horizon proactive assistance. ProMemAgent organizes past sensory perceptions (e.g., egocentric video and audio) into context-aware memory, generating prospective cues and linking earlier events with subsequent developments for future retrieval. Given the current context, it retrieves relevant past events and performs memory-augmented proactive reasoning to determine whether assistance is needed. We further introduce CPMBench, the first benchmark for evaluating long-horizon proactive assistance over sensory contexts, comprising 1,000 instances across eight everyday categories. Extensive experiments show that ProMemAgent achieves the best memory retrieval performance and consistently outperforms all baselines on long-horizon proactive assistance.