Transformer combines multiple camera views for better 3D human pose estimates
STA-TFM: Spatio-Temporal Aggregation Across Views TransForMer for Pose Estimation
Computer Vision and Pattern Recognition
Summary
Estimating 3D human poses from a single camera is hard because the camera can only see from one angle, making it tricky to understand depth and handle blocked views. The authors created a new method called STA-TFM that uses a transformer model to combine information over time from several cameras to make more accurate 3D pose estimates. They also developed a way to create training data with different camera setups to help the model learn. Their method works better than other approaches that don’t rely on camera settings and handles missing or noisy input well.
What this means in practice
- •For healthcare technology developers: Improve accuracy of patient movement tracking in healthcare monitoring systems using multi-view 3D pose estimation without complex setup.
- •For sports performance analysts: Analyze athletes’ motions with reliable 3D pose estimation using fewer cameras and better temporal consistency.
- •For game engine developers: Integrate robust multi-view pose estimation modules that handle noisy or missing data for immersive virtual reality experiences.$Commercial implications: Enables development of VR systems that track user movement accurately from multiple cameras, improving immersion and interaction.
Authors
Mena Kamel, Natalie Won, Amrut Sarangi, Sven Jager, Albert Pla Planas
Abstract
Monocular 3D human pose estimation (HPE) remains challenging due to depth ambiguity, occlu- sions, and the need for temporal consistency. While multi-view methods provide superior accuracy over monocular approaches, they often require complex setups. We introduce STA-TFM, a transformer-based architecture that combines spatial and temporal information for multi-view pose estimation. The approach leverages DSTformer, a monocular feature extractor, to capture long-range pose dependencies within each view. A fusion transformer then aggregates information across views to produce coherent 3D estimates. To address training data scarcity, we use a data generation pipeline that transforms any existing 3D pose dataset into multi-view setups with controllable parameters. Experiments on various datasets demonstrate that STA-TFM outperforms existing camera-parameter-free multi-view methods. STA-TFM achieves 50.9% and 49.5% reductions in mean per joint position error (MPJPE) and mean per joint velocity error (MPJVE) on the DHP19 dataset. Furthermore, it achieves 6.7% and 7.7% respective reductions on HAA4D, and a 15.2% MPJPE reduction on TotalCapture. STA-TFM handles noisy and missing 2D inputs, supporting potential deployment in healthcare monitoring, athletic assessment, and immersive technologies. Code, training checkpoints, and data are available at https://zenodo.org/records/22832620.