Foundation models detect climbers hand and foot holds accurately
Training-Free Hold-Usage Detection in Sport Climbing with Foundation Pose Models
Computer Vision and Pattern Recognition
Summary
Sport climbing needs to know when and where climbers touch holds for scoring and analysis. The authors show that using a general pose model without extra training can find which holds climbers use by checking how close fingertips and toes are to the holds. Their simple method works better than other complex models, especially for feet, and matches up well with real coaching stats. This means regular climbing videos can be turned into useful performance data without special equipment.
What this means in practice
- •For sport climbing app developers: Build automatic climbing performance tracking features that detect hold usage from video without extra training data or sensors.$Commercial implications: Enables development of consumer climbing apps with automatic hold usage scoring using only single-camera video input.
- •For physical therapy clinics: Use off-the-shelf pose models to monitor climbing patients’ movements and foot placement from standard videos for rehabilitation feedback.
Authors
Abu Bakar, Abdullah Aftab, Amir Hamza
Abstract
Detecting which holds a climber uses, and when, underpins automated scoring, movement analysis, and assistive systems for sport climbing. Existing approaches train task-specific models or repurpose 2D pose estimators whose hand keypoint sits at the wrist and foot keypoint at the ankle i.e. offset from the fingertips and toes that actually contact the holds, and whose hands are occluded in roughly half of all frames. We show that a frozen, off-the-shelf pose foundation model is sufficient: using the fingertip and toe keypoints of Sapiens, a per-frame proximity test against the annotated holds, per-limb mutual exclusion, and a short temporal-persistence rule, we detect hold usage without any climbing-specific training. On the The Way Up dataset (22 videos, 10 athletes, two routes), our method reaches an event F_1 of 90.2% on a held-out split (89.8% under leave-one-participant-out cross-validation) and 79.9% over all 22 videos at any temporal overlap, and performs best on footholds (F_1,89.8% overall, 96.6% held-out). Under an identical protocol it exceeds our reproductions of the YOLOv8-pose and ViTPose pipelines at every temporal threshold, with the margin widening under strict timing. An ablation shows that two intuitively helpful additions---dense foundation-feature change gating and body-part segmentation---both hurt, arguing that a minimal, keypoint-only design is the right one for this task. Finally, standard coaching statistics computed from our automatic predictions track ground truth closely (Pearson r=1.00 for climb time, 0.94 for pace), turning ordinary single-camera video into reliable performance metrics with no instrumentation.