Universal model learns effective motion patterns from all skeleton sensors
One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning
Computer Vision and Pattern Recognition
Summary
Different devices that track human movement use different kinds of skeleton models that don’t always match in shape or joint numbering. This makes it hard to build one model that works well across all these devices. The authors created a new model called SOfA that can understand skeleton data from many types of sensors by using a fixed set of ‘slots’ to capture key points and by using language-based joint descriptions for alignment. They trained it on ten different datasets and showed it works well on many tasks, often better than models trained for just one type of sensor.
Self-supervised learningSkeleton representationCross-sensor dataAttention mechanismCanonical Joint SlotsSemantic Joint Embedding3D skeleton trackingFoundation modelMotion representationMultimodal learning
Authors
Jeonghyeok Do, Yun Chen, Munchurl Kim
Abstract
For learning generalizable motion representations from large-scale unlabeled data, Self-supervised learning (SSL) has become a widely adopted methodology. However, existing approaches are primarily limited by the inherent heterogeneity of skeleton data---characterized by varying joint counts, indexing protocols, and topological structures across different sensors---which typically necessitates training separate, sensor-specific, or even entirely dataset-specific models. To overcome this, we introduce SOfA (Skeleton One for All), the first generalist foundation model designed to achieve sensor-unified skeleton representation learning across diverse sensors. To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots, acting as a universal vessel that seamlessly accommodates arbitrary skeletal topologies. SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs. Furthermore, we resolve joint index misalignment between various sensors by introducing a Semantic Joint Embedding derived from a pre-trained text encoder, rather than relying on absolute positional embeddings. To validate our approach, we standardized ten 3D skeleton datasets for unified training. Extensive experiments demonstrate that SOfA can serve as a truly universal encoder, achieving state-of-the-art (SOTA) performance across a wide range of downstream tasks and sensor types, often outperforming dataset-specific specialist models with a single foundation model.