Hierarchical step decomposition improves online video step recognition and timing
Hierarchical Step Decomposition for Training-free Online Video Step Grounding
Computer Vision and Pattern Recognition
Summary
Sometimes we want a computer to watch a video and tell us exactly when certain steps or actions happen, like following a recipe or instructions. Existing methods either need to see the whole video first or look at fixed chunks, which can be noisy. The authors created a new way, called Hierarchical Step Decomposition, that watches videos live and groups events more naturally without training. It uses large AI models to match video content to steps, breaking down complex steps into smaller parts for better accuracy. This method is faster, stable, and works better at recognizing and timing steps in live videos.
What this means in practice
- •For video surveillance teams: Identify and timestamp procedural steps from live surveillance feeds for clearer event understanding.
- •For industrial process monitors: Automatically track and localize steps in live manufacturing videos to detect deviations or inefficiencies early.
Authors
Long Zhou, Luca Zanella, Paolo Rota, Massimiliano Mancini, Elisa Ricci
Abstract
Given a set of procedural steps, Video Step Grounding (VSG) aims to identify which steps occur in a video and localize when they are performed. Existing methods are either offline, requiring access to the full video before making predictions, or online, but relying on fixed-length video segments producing noisy, over-segmented outputs. In this work, we propose Hierarchical Step Decomposition (HSD), a training-free framework for online VSG that leverages the semantic understanding capabilities of Large Multimodal Models. HSD addresses two central challenges in online grounding: constructing semantically coherent temporal segments from a streaming video, and bridging the semantic gap between coarse procedural step descriptions and visual observations. To construct elementary temporal units, HSD incrementally groups incoming segments into semantically consistent events, using an anchor caption that is continuously validated against the visual stream. Based on these events, we perform zero-shot procedural step classification and selectively refine uncertain predictions by decomposing candidate steps into finegrained substeps that capture more nuanced visual evidence. By operating on semantically coherent events rather than every fixed-length segment, HSD reduces redundant computation and produces more stable predictions. Experiments on HT-Step, Ego4D Goal-Step, and CrossTask show that HSD substantially outperforms state-of-the-art methods, overall achieving better step recognition, more accurate temporal localization, and faster inference.