Multimodal 3D human pose estimation with built in privacy safeguards
Kinematics-Induced Multimodal 3D Human Pose Estimation with Subject-Level Privacy
Computer Vision and Pattern Recognition
Summary
Estimating 3D human poses from multiple sensors like cameras, LiDAR, and radar helps computers understand movements better. The authors created a system that combines these data sources while protecting individual privacy by making it hard to identify people in the data used for training. They also developed methods to detect privacy risks and trained their model with techniques that keep subject information private. Their approach was tested on a real dataset to show good results.
What this means in practice
- •For autonomous vehicle developers: Improve pedestrian motion prediction by fusing privacy-respecting sensor data from cameras, LiDAR, and radar for safer navigation.
- •For smart building engineers: Deploy privacy-aware multimodal sensing for monitoring occupant activities without compromising individual identities.
Authors
Kaushik Bhargav Sivangi, Fani Deligianni
Abstract
Multimodal 3D Human Pose Estimation (3D HPE) combines complementary information from RGB, LiDAR, and mmWave radar, but models trained on correlated observations from the same individuals, raise privacy risks overlooked by record level analysis. We present a unified framework for multimodal 3D HPE that couples kinematics-induced sensor fusion with subject level privacy auditing and private training. First, our multimodal model aligns modality specific joint representation, injects skeletal structure and adaptively aggregates complementary sensor evidence for accurate pose prediction. Second, we formulate a black-box subject membership inference attack for 3D HPE, complemented by an empirical pointwise maximal leakage analysis, which characterizes how individual attack score outcomes change inference about the membership outcome. Third, we instantiate user-level differential privacy via Action Temporal Stratification, a population weighted within-subject sampling strategy that enforces action and temporal coverage. We evaluate our framework on the MM-Fi dataset across three diverse experimental protocols. Source-code will be released upon acceptance.