DeformStruct improves action timing with deformable structure modeling
DeformStruct: Deformable Structure Modeling for Point-Supervised Temporal Action Localization
Computer Vision and Pattern Recognition
Summary
Finding the exact start and end time of actions in videos is hard when you only have one timestamp as a clue. The authors created DeformStruct to better understand the parts of each action and how they fit together over time. Their method finds key points, matches action parts that can shift around, and models how those parts relate to each other. This helps their system locate actions more precisely using limited time annotations and works well on several action video datasets.
What this means in practice
- •For video content analysts: Enable more accurate action timing detection from videos when only sparse timestamp labels are available, improving workflow efficiency.
- •For surveillance system engineers: Improve event detection in security footage by better modeling how actions unfold over time using minimal annotation effort.
Authors
Mengnan Liu, Sanping Zhou, Kun Xia, Yonghao Dong, Jie Wang, Gang Hua
Abstract
Point-supervised Temporal Action Localization (PTAL) aims to localize action instances using only a single annotated timestamp for each instance. Such sparse supervision offers limited guidance for modeling the structural organization of an action instance. Existing PTAL methods primarily infer action regions from classification responses, leaving the structural organization of multiple action-related parts within each instance insufficiently modeled. To address this problem, we propose DeformStruct, an instance-level deformable structure modeling framework for PTAL. DeformStruct progressively constructs the structure of each action instance through three stages. First, Structural Anchor Mining discovers multiple anchors from annotated points and action-responsive regions, providing structural nodes grounded in the current instance. Second, Deformable Part Alignment associates latent structural parts with these anchors through deformation-aware matching, allowing their temporal supports to adapt to instance-dependent variations. Third, Relational Structural Organization explicitly models the relative temporal relations among the aligned parts, forming a relational representation that characterizes how different action-related parts are organized together. Extensive experiments on THUMOS14, ActivityNet-v1.3, GTEA, and BEOID demonstrate consistent improvements in point-supervised temporal localization and validate the effectiveness of deformable structural modeling.