Wearable AI decides when to interrupt based on video analysis
Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
This work tackles how a wearable AI assistant can tell if it should interrupt its wearer after watching short videos from their viewpoint. The authors improved prediction by changing the task to a simple yes-or-no decision instead of more complex responses. To overcome limited labeled examples, they used a video agent to label more data with timings for when the AI should intervene. They found that adding visual context in training data is more helpful than just increasing the amount of text-based labels.
What this means in practice
- •For wearable device developers: Build smarter wearable assistants that decide when to interrupt users by analyzing first-person video segments more accurately.$Commercial implications: Enables creation of more responsive and less intrusive assistive wearables for consumers, improving user experience and engagement.
- •For video annotation teams: Use visually grounded video agents to generate precise intervention timing labels, reducing reliance on costly manual labeling.
Authors
Logesh Kumar Umapathi
Abstract
We present our submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, which ranked first in the large-model division and second in the <=2B division. The task requires a wearable assistant to decide after each eight-second segment of egocentric video whether to intervene or remain silent. Our approach has two main components. First, we reformulate intervention timing as single-token classification. Rather than generating either $interrupt$<utterance> or $silent$, the model predicts yes or no, and we derive the decision from the renormalised probabilities of these two tokens. This formulation improved macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Second, because labelled data were limited to the released validation set, we generated additional supervision using a tool-calling video agent that inspects each clip and assigns intervention timestamps. A narration-only alternative was four times larger and ten times cheaper, but transferred worse than supervision from an unrelated real corpus, suggesting that visual grounding is more important than annotation volume for this task.