Surgical video models improve by respecting procedure hierarchies
A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Surgical videos show operations that happen in steps grouped into larger phases, but current video-language tools treat these steps equally without considering their hierarchy. The authors created a new way to check how well models honor this structure by measuring not just overall accuracy but also consistency and error severity across these levels. They also developed a new type of model that uses hyperbolic geometry, which better captures the tree-like hierarchy of surgeries. Their tests show this approach reduces serious mistakes and works better on datasets where the hierarchy is clearer.
What this means in practice
- •For medical ai developers: Enhance surgical procedure recognition systems by ensuring predictions respect phase-to-step hierarchical structures, reducing error severity.
- •For industrial ai teams: Improve video understanding models for complex processes by integrating hierarchy-aware hyperbolic geometry to better handle structured task sequences.
Authors
Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven, Cees G. M. Snoek
Abstract
Surgical procedures follow a phase-to-step hierarchy, yet the video-language models used to recognize them are evaluated with flat per-level metrics that ignore cross-level coherence and error structure. In this paper we make two contributions to address this problem, (i) we introduce SurgHiBench, the first hierarchy-aware evaluation suite for surgical video understanding, with three tasks measuring recognition, consistency, and severity across granularity levels. We evaluate a general-purpose CLIP model, a Euclidean surgical model, and, as second contribution: (ii) HyperSurg, a new hyperbolic model that enforces phase-step containment via entailment cones, across four (existing) datasets spanning three procedure types. The suite reveals that two models with the same accuracy can produce predictions of very different error severity, ranging from sibling confusions within the correct phase to unrelated cross-phase predictions. Hyperbolic geometry shifts predictions toward the correct procedural neighborhood, and these gains scale with the tree-likeness of each dataset's annotation hierarchy, providing a principled indicator when hierarchy-aware geometry helps.