Joint method improves 3D face scan alignment and shape completion
Joint Rigid Alignment and Morphable Reconstruction from Partial 3D Face Scans
Computer Vision and Pattern Recognition
Summary
When scanning faces in 3D, sometimes the scan only captures part of the face and is not aligned properly. The authors developed a method that does two things at once: it figures out how to rotate and position the scanned face correctly and fills in the missing parts of the face shape. Their approach uses a neural network trained on many faces to predict a complete face model that best fits the partial scan. This method is faster and more accurate than previous ones and does not need detailed manual labels for training.
What this means in practice
- •For 3d graphics developers: Create more accurate and fast 3D face reconstructions from partial scans for use in animation and virtual avatars.
- •For biometric security teams: Improve face recognition systems by better aligning partial 3D face data to standardized models for more reliable identification.
Authors
Eli A. Underwood, Nicolas Rondan, Angel V. Peterchev, J. Matias Di Martino
Abstract
Aligning a partial, real-world 3D face scan to a canonical representation requires solving two coupled problems at once: estimating the rigid pose that brings the scan into a common frame, and completing the facial shape in regions the scan never observed. Existing pipelines typically decouple these steps, running a generic point-cloud registration front end followed by a separate non-rigid fitting or deformation stage. Building on the GeoTransformer superpoint-matching architecture, we propose a single network that predicts both a rigid pose and a shape-complete point cloud in one forward pass: a cross-attention regressor, conditioned on transformer point features and predicted coarse correspondences, maps a partial scan to per-patch coefficients of a 3D morphable face model learned offline from a corpus of 8,000 identities. The resulting morphed template is then used, in the same forward pass, as the target for fine-grained Sinkhorn matching and rigid pose estimation, supervised purely by geometric alignment to ground-truth morphed geometry, so no hand-labeled coefficients are required. Against six classical and learned non-rigid registration baselines on held-out identities, our method leads the strongest ones by 3-4 times on per-vertex error and baselines that collapse the head onto the visible region by more than an order of magnitude, and is simultaneously the most accurate and fastest method on ground-truth-free scans from two independent real-scan datasets, NoW and NPHM. Code will be made publicly available.