Human mesh recovery improves with privacy-safe local training methods
PCHMR: Empowering and Benchmarking Human Mesh Recovery in Privacy-Constrained Real-World Settings
Computer Vision and Pattern Recognition
Summary
Human mesh recovery means creating a 3D model of a person from images, which helps in many tech applications. Usually, models get better by collecting lots of user images on a central server, but this raises privacy concerns. The authors study how to train and evaluate these models without sharing raw images, using methods that keep data on users’ devices. They also introduce a new way to fine-tune models locally to better fit individual users’ data, improving accuracy while respecting privacy.
What this means in practice
- •For mobile app developers: Build apps that create personalized 3D human models without uploading user photos to servers, maintaining user privacy.
- •For security software teams: Develop privacy-preserving biometric systems that adapt locally to users while avoiding centralized storage of sensitive images.
Authors
Zeyu Cao, Qingxuan Wu, Zhiyang Dou, Rui Xu, Yuan Liu, Javier Fernandez-Marques, Nicholas D. Lane, Taku Komura, Wenping Wang
Abstract
Fine-tuning directly on user-side data is an effective way to improve the performance of widely adopted data-driven human mesh recovery (HMR) models. However, doing so for existing HMR methods often assumes aggregating user-side data from real-world HMR-deployed devices to a central training server, posing a significant privacy risk as sensitive human images are transmitted. How can human mesh recovery be evaluated and improved in privacy-constrained real-world settings? We study this question through a benchmark in which state-of-the-art HMR models are trained with federated learning and a secure-aggregation variant, neither of which centralizes raw images, and evaluated across a wide range of client-scale and data-heterogeneity settings. We document that common HMR training pipelines are built around centralized data access and quantify how representative HMR backbones behave when that assumption is removed. Furthermore, to study the data bottleneck of privacy-constrained HMR training, we propose DePoser, a local annotation and fine-tuning pipeline enhanced with depth foundation models, with which collaboratively trained HMR models can be locally tailored to the end user's distribution. On in-the-wild data, fine-tuning on DePoser annotations improves personalization accuracy, measured against DePoser-generated pseudo-ground truth. This benchmark aims to support future work on privacy-constrained HMR models and their real-world deployment and evaluation.