Egocentric video question answering improved by targeted textual memory retrieval
Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering
Computer Vision and Pattern Recognition
Summary
Answering questions about long videos from a first-person perspective is hard because the video’s story is too long to remember all at once. The authors studied how picking out the right pieces of text summaries from the video can help computers answer questions better. They found that showing only a few relevant, complete text parts works better than just grabbing more text blindly. This means smarter ways to remember and retrieve video details can improve how computers understand and answer questions about experiences.
What this means in practice
- •For video app developers: Create apps that answer questions about user-recorded first-person videos by retrieving key text summaries efficiently.
- •For surveillance analysts: Assist in reviewing long security videos by surfacing concise, relevant textual memories to quickly answer questions about events.
Authors
Raffaele Cali, Giuseppe Lando, Rosario Forte, Antonino Furnari
Abstract
Long egocentric videos pose a central challenge for multimodal language models: the visual stream must be compressed into a memory representation that can be searched efficiently while still preserving the evidence needed for question answering. Building on an online textual-memory framework in which first-person videos are converted into temporally grounded captions, this work systematically studies the role of retrieval in episodic-memory video question answering. We evaluate how different retrieval units, query formulations, context budgets, and reranking strategies affect multiple-choice answering on QAEgo4D-Closed, isolating retrieval as the main variable while keeping the underlying generated memory fixed. Our best configuration, Parent, matches fine-grained caption subchunks, then provides their full parent captions to the reasoner. It achieves 56.4 % accuracy, matching strong full-memory and retrieval baselines while exposing only a small fraction of the textual memory at question time. More importantly, our analysis shows that better temporal coverage does not necessarily translate into better answers: increasing the retrieval budget improves temporal Hit@K and Recall@K, but can reduce accuracy by introducing visually plausible distractors and uncertain caption evidence. These findings suggest that retrieval-augmented egocentric memory should not be optimized only for evidence recall, but also for the reliability, specificity, and answer-discriminative value of the retrieved context. The study motivates more structured episodic memories that move beyond raw caption retrieval toward evidence-aware representations for long-form egocentric reasoning.