Procedural mistake recognition improves by linking video to language

Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos

Computer Vision and Pattern RecognitionComputation and Language

Summary

Recognizing mistakes in multi-step tasks from videos is hard because it requires checking if each step was done correctly and in order. The authors introduce PRISM, a system that describes video steps in language, then uses a model trained on correct examples to find errors. They show that careful use of language descriptions helps spot mistakes across different datasets. This approach improves detecting various types of errors like missing steps or steps done in the wrong sequence.

What this means in practice

  • For assembly line quality teams: Identify errors in step-by-step assembly videos by matching video content to expected procedure descriptions to improve quality checks.
  • For video content analysts: Automatically detect mistakes and out-of-order steps in how-to or instructional videos by aligning language descriptions with video segments.

Authors

Olga Loginova, Oleksandr Bezrukov, Frank Keller

Abstract

Procedural mistake recognition requires identifying deviations from valid multi-step activities. We study post-completion recognition from videos with benchmark-provided step boundaries. We introduce PRISM (Procedural mistake Recognition via Interpretable Semantic Modeling), which grounds video segments in language and maps their descriptions to procedural states using a lightweight hidden Markov model trained on correct executions. Reference alignment and semantic checks identify errors in step content and order. To support evaluation across heterogeneous benchmarks, we map dataset-specific annotations to four edit-based error types: insertion, deletion, substitution, and transposition. Across three benchmarks, we compare gold and VLM-generated descriptions in raw, semantic-role, and action-object forms. Decoder controls show consistent gains from the trained local decoder over nearest-candidate matching, with additional, dataset-dependent benefits from sequence decoding. Semantic role labeling improves Any-F1 in all six matched grounded comparisons, although the gains vary across datasets and do not translate into uniform improvements under separate configuration selection. These results establish language grounding as an effective interface for explicit procedural mistake reasoning and characterize the contributions of representation and decoding.