Augmented reality helps fencers find and fix movement mistakes
FenceXR: AR Movement Replay for Error-Detection Training and Spatially Grounded Feedback
Human-Computer Interaction
Summary
Spotting mistakes in fast sports moves like fencing is hard for beginners and coaches using only videos or live demos. The authors created FenceXR, an augmented reality tool that shows 3D replays of fencing moves from simple phone videos. Beginners use it to practice recognizing common errors, and coaches can attach notes to specific parts of the replay to guide learners. Tests showed novices got much better at spotting errors after just one session, and experts found the tool useful for giving detailed feedback.
What this means in practice
- •For sports coaches: Provide trainees with 3D movement replays and spatially anchored feedback to improve error detection in complex sports training.
- •For physical therapy teams: Use AR-based movement replays to monitor patients’ motions and offer precise, joint-specific feedback to aid rehabilitation exercises.
Authors
Avinash Ajit Nargund, Amelia Haruka Harrison, Timothy Robinson, Tobias Höllerer, Misha Sra
Abstract
Recognizing technical errors in movement is a perceptual skill important to motor learning, but it is challenging for beginners in fast, complex sports like fencing to develop it. A coach's attention is scarce, live demonstrations vary from repetition to repetition, and video review is limited to whatever camera angle was used to record it. Coaches and advanced fencers reviewing a recording face a related problem. They can see an error, but have no way to anchor their feedback to the movement itself, and are left describing it in words the learner must map back onto their own body. We present FenceXR, an augmented reality system that reconstructs 3D movement replays from monocular smartphone video to address both problems. A Trainee module trains novices to detect common lunge errors while a Reviewer module lets coaches and advanced fencers attach text or voice annotations to a specific joint and moment in a replay, which can be shared asynchronously with a trainee. In a study with 18 novice fencers, unaided error-detection accuracy rose from near-chance (37.5%) before training to 64.1% after a single session, with interviews showing a shift from broad visual scanning toward targeted inspection of specific joints and their timing. In a video-based study with four fencing experts, all four viewed the Reviewer module as a valuable complement to their existing coaching tools, particularly for feedback that is difficult to convey through standard video. We end with a discussion of implications for designing AR systems that ground movement-based training and feedback in the movement itself.