Prototype guided pretraining improves human sense in mmWave point clouds
PT$^3$: Prototype-Guided Self-Supervised Pretraining for Generalizable mmWave Human Point Cloud Representation
Artificial Intelligence
Summary
Millimeter-wave sensors can detect human shapes and movements but often produce messy, incomplete data that makes it hard to understand what’s going on. The authors created a new way to train computers to recognize human figures from these imperfect signals by teaching them common building blocks and how they change over time. This training method helps the computer better handle noisy data and works well across different tasks, like recognizing actions or predicting motion, even on data it hasn’t seen before. This means the system can learn from unlabeled data and adapt to many human sensing jobs.
What this means in practice
- •For robotics developers: Create robots that understand human pose and actions from radar point clouds across different environments with less labeled data.
- •For security system integrators: Develop sensing systems that detect and predict human motion in privacy-sensitive environments using mmWave radar data robust to noise.
Authors
Ziyang Liu, Kieren Yu, Xiao Yang, Xin Yang, Kaishun Wu
Abstract
Multi-source millimeter-wave (mmWave) observations offer an opportunity to learn human representations reusable across sensing tasks. However, sparse detections and multipath effects make these observations unstable, challenging existing point cloud pretraining methods. We propose PT$^3$, a prototype-guided self-supervised pretraining framework for mmWave human point clouds. PT$^3$ represents local states in spatial regions as sparse compositions of shared prototypes and learns their temporal evolution through autoregressive prediction. Observation consistency further stabilizes representations under radar perturbations. After pretraining on unlabeled sequences from multiple sources, the same frozen encoder supports action classification, dense point generation, joint pose estimation, and future motion prediction on datasets not used for pretraining. Across these tasks, PT$^3$ outperforms the compared pretraining baselines and achieves competitive or superior performance to task-specific models.