Wearable AI learns when to give helpful advice using video context
Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The problem is teaching a wearable AI assistant when to speak up or stay quiet based on what it sees through a user's camera. The authors improved this by turning the decision into a simple yes-or-no choice, which made the assistant more accurate. Because there was limited labeled data, they created extra training examples by having a video agent watch clips and suggest the best times to intervene. They found that using visual cues worked better than just relying on written descriptions, even if there was more text data available.
egocentric videowearable AIintervention timingsingle-token classificationmacro-F1 scorevisual groundingsupervised learningtool-calling video agentdata augmentationmodel evaluation metrics
Authors
Logesh Kumar Umapathi
Abstract
We present our submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, which ranked first in the large-model division and second in the <=2B division. The task requires a wearable assistant to decide after each eight-second segment of egocentric video whether to intervene or remain silent. Our approach has two main components. First, we reformulate intervention timing as single-token classification. Rather than generating either $interrupt$<utterance> or $silent$, the model predicts yes or no, and we derive the decision from the renormalised probabilities of these two tokens. This formulation improved macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Second, because labelled data were limited to the released validation set, we generated additional supervision using a tool-calling video agent that inspects each clip and assigns intervention timestamps. A narration-only alternative was four times larger and ten times cheaper, but transferred worse than supervision from an unrelated real corpus, suggesting that visual grounding is more important than annotation volume for this task.