Caption memory improves long video understanding for wearable assistants
CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Understanding and remembering what happens in long first-person videos is hard for current AI systems, especially with limited computing power. The authors study if using short text captions as memory can help AI better answer questions about these videos. They created a new set of videos and questions to test this idea and found that caption-based memory helps models answer questions more accurately than just analyzing the video frames. This suggests that captions can be a useful way to help wearable devices remember and reason about long experiences.
What this means in practice
- •For wearable device developers: Enhance wearable assistants to recall and reason about long user experiences using caption-based episodic memory.$Commercial implications: Enables production of smarter wearable devices that efficiently remember and interpret first-person video for user assistance.
- •For video analytics teams: Use caption-guided retrieval to improve question answering accuracy over long-duration egocentric videos without exhaustive frame processing.
Authors
Dingli Liang, Yiqiao Xie, Yukai Huang, Zhaokai Wang, Weitong Cai, Guangwen Feng, Jifei Song, Zhensong Zhang, Hang Zhang
Abstract
Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic memory. We define the Episodic Memory Video Caption QA task and introduce CapMem, a human-annotated benchmark with 75 videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long videos (>20 min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption memory for episodic reasoning over long egocentric video.