Semantic adaptive method improves 3D avatar motion and lighting effects
Relightable 3D Avatar Reconstruction with Semantic-Adaptive Motion-Illumination Responses
Computer Vision and Pattern Recognition
Summary
Making realistic 3D animated faces from a single video is hard because faces move in complex ways and look different under various lighting. The authors created a new way called SAMIRA that treats different parts of the face separately, so it better captures how each area moves and reflects light. This helps the avatars show detailed expressions and respond more naturally to changing lighting. Tests show SAMIRA is better at copying facial movements and making relighting look real compared to older methods.
What this means in practice
- •For game developers: Create more expressive and realistically lit 3D character faces from simple camera footage for games.$Commercial implications: Enables production of high-quality 3D avatar animations with realistic light responses, enhancing game character realism.
- •For film production teams: Generate accurate facial animations with nuanced lighting for digital doubles from standard video captures in film VFX pipelines.
Authors
Jiankuo Zhao, Xiangyu Zhu, Jijie Li, Baiqin Wang, Shukai Chen, Zhen Lei
Abstract
Reconstructing expressive and relightable 3D head avatars from monocular videos remains challenging in computer vision, as it requires accurate modeling of both non-rigid facial motion and illumination-dependent appearance. Existing Gaussian avatar methods commonly rely on globally coupled representations, in which Gaussian primitives share a unified motion or illumination response model. Such uniform modeling neglects the distinct motion patterns and material/reflectance properties of different facial semantic regions, thereby limiting fine-grained animation accuracy and reducing relighting plausibility. To address this limitation, we propose SAMIRA, a 3D Gaussian avatar framework for semantic-adaptive motion-illumination response modeling. For motion response modeling, the Semantic-Adaptive Motion Response module rasterizes current-to-reference mesh displacements into a topology-consistent UV space and leverages facial semantics to route displacement features through semantic-specific modulators, predicting localized Gaussian geometric residuals beyond coarse mesh binding. For illumination response modeling, the Semantic-Adaptive Illumination Response module learns compact diffuse and specular response factors for each facial region, allowing Gaussians in different regions to adapt their illumination responses to novel environment lighting. These response factors are incorporated into deferred physically based shading, providing a lightweight approximation of semantic-dependent illumination effects. Extensive experiments on self-reenactment, cross-reenactment, and relighting demonstrate that SAMIRA improves both fine-grained expression reconstruction and relighting realism over existing methods.