Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
2026-06-08 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors focus on improving how smart video assistants can help people learn tasks like cooking by spotting and correcting mistakes right away. They created a new test called Ego-MC-Bench to see how well current video language models handle real cooking guidance step-by-step, finding it quite hard. Because there isn't much training data showing mistakes and timely corrections, the authors also made a new synthetic dataset called Ego-CoMist that adds these examples. Training models with Ego-CoMist helped smaller, efficient models get better at giving helpful advice, which is useful for devices like phones or smart assistants.
video large language modelstask guidanceproactive interventionmistake correctionEgo-MC-BenchEgo-CoMistcounterfactual datasetfine-tuningcooking video datasetsedge devices
Authors
Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh, Rajeev Yasarla, Reza Pourreza, Litian Liu, Risheek Garrepalli, Roland Memisevic
Abstract
Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) large language models (LLMs) as task guidance assistants. A crucial capability for the real-world success of a prospective task guidance assistant is it's ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a benchmark for evaluating reactive, step-by-step task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state-of-the-art video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking video datasets, existing datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic dataset created by transforming non -interactive cooking videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient video LLMs that are well suited for delivering assistance on edge devices.