Real-time hand gesture recognition improved with transformer models
Real-Time Hand Gesture Recognition for OpenXR Using Transformer-Based Machine Learning
Artificial Intelligence
Summary
Hand gesture recognition helps computers understand what your hands are doing so you can interact naturally in virtual reality, games, or with robots. The authors built a system that uses a special kind of AI model called a transformer to look at sequences of hand positions over time. This approach helps the system recognize gestures more accurately, even if hands are different sizes or orientations. They tested it using hand-tracking data from a widely used VR framework called OpenXR.
What this means in practice
- •For virtual reality developers: Integrate a more accurate real-time hand gesture recognition system into VR applications using OpenXR data.
- •For robotics engineers: Use transformer-based gesture recognition to improve robot control interfaces involving hand signals.
Authors
Salar Rezayani, Russell Butler
Abstract
Hand gesture recognition is a key component in human-computer interaction (HCI), enabling intuitive interfaces for applications in gaming, virtual reality (VR), robotics, and more. This study integrates transformer-based machine-learning models for real-time hand gesture recognition, using hand-tracking data captured through the OpenXR standard in Unity. We leverage positional data of hand joints and wrist rotation angles to train a custom gesture recognition system. By utilizing the sequential modeling capabilities of transformers, the system captures temporal dependencies within short gesture windows and classifies gestures robustly across hand orientations and sizes. The results show a significant improvement in gesture classification accuracy. Building on this, we outline how the approach can be extended toward detecting the flow of movement, i.e., the transitions between gestures, as future work.