Benchmark evaluates proactive memory for wearable video assistants
ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video
Computer Vision and Pattern Recognition
Summary
Remembering important details over long periods is hard for wearable assistants. This paper presents a benchmark called ProactEgoMem that tests how well assistants can proactively capture and track information from videos recorded by users wearing cameras. It includes tasks like remembering fleeting information, object states, and locations across daily and scientific activities. The benchmark also measures how timely and accurate these assistants are when they notify users or actively maintain memory. The tests show that current models still struggle to do this well.
What this means in practice
- •For wearable device developers: Evaluate and improve memory capabilities in smart glasses and other wearable video assistants during real-time user activities.$Commercial implications: Makes it possible to build more proactive wearable assistants that anticipate user needs via memory, improving user experience and device usefulness.
- •For video processing software teams: Test algorithms that track dynamic changes and capture critical moments in continuous video streams from a first-person viewpoint.
Authors
Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun, Yongqin Xian, Laura Bravo Sánchez, Shiye Su, Federico Tombari, Serena Yeung-Levy, Volker Tresp
Abstract
Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.