VidParse: Online Parsing of Egocentric Procedures Like a Pro
Computer Vision and Pattern Recognition
Summary
The authors address the problem of understanding continuous first-person videos by breaking them into meaningful action steps. They propose VidParse, a method that doesn’t need training and uses a graph to guide which actions can follow others. Instead of relying on typical frame-by-frame analysis, they use similarity between video parts and known task structures to better recognize the sequence of actions. This approach improves accuracy significantly in complex, multi-step activities without needing to update any model weights.
Authors
Anubhav Gupta, Archit Kambhamettu, Vatsal Agarwal, Pulkit Kumar, Abhinav Shrivastava
Abstract
Translating continuous, noisy egocentric video streams into discrete, temporally ordered action steps is fraught with visual challenges. Heavy ego-motion, transient occlusions, and the high intra-class variability of unscripted human-object interactions cause standard frame-level online temporal models to struggle, often resulting in severe over-segmentation and structural collapse. To bridge the gap between unstable low-level perception and high-level procedural logic, we present VidParse, an online, training-free framework that treats activity understanding as a graph-constrained inference problem. Rather than relying on learned temporal filters, we dynamically identify semantic transitions using a temporal similarity matrix over manipulation-anchored features, which are extracted from frozen foundation models to prioritize foreground hand-object interactions. A beam search decoder then leverages an induced procedural task graph to explicitly enforce valid action transitions and prune impossible trajectories. By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10x improvement in complex multi-step parsing accuracy over strong online baselines, all without requiring a single gradient update.