Geometry based method improves shadow realism in face swaps
Geometry-Driven Shadow Harmonisation for Composited Faces: A Multiplicative, Albedo-Preserving Relighting Pipeline
Computer Vision and Pattern Recognition
Summary
Face swapping often makes faces look out of place because lighting is different between the face and the background. Existing fixes can change skin color or identity, which is not ideal. The authors propose a new way to fix shadows realistically by using a 3D face model to darken parts of the face without changing its color or texture. This method adapts shadows to match the lighting direction of the background more naturally without repainting the face. They tested it and explain how the approach works, its limits, and why it keeps skin color unchanged.
What this means in practice
- •For visual effects artists: Improve shadow consistency in composited faces to create more natural visuals without altering skin tone or identity.
- •For mobile app developers: Enhance face swap apps by implementing geometry-driven shadow adjustments that keep skin color intact under varied lighting.
Authors
Vijesh KP
Abstract
Face swapping and face compositing pipelines routinely produce a face that is geometrically well aligned but photometrically implausible: the donor face carries flat, near-frontal studio illumination while the host body and background carry directional scene light. Most existing remedies re-synthesise the face through colour transfer, neural relighting, or inverse rendering, and therefore risk altering identity, skin tone, and texture. We present a conservative alternative: geometry-driven form-shadow injection. The pipeline never repaints the face. It estimates a per-pixel gain field $g\in[g_{\min},1]$ from a rasterised 3D face proxy and multiplies it channel-uniformly onto linear RGB, so the operator can only darken and cannot shift chromaticity. A dense landmark mesh is rasterised into a depth buffer, from which we derive surface normals, a cavity term, and screen-space cast shadows. Key-light direction is estimated from host-side cues (body, background, hair halo); on-face cues are downweighted because they recover the donor's lighting. Shadow magnitude is not matched to the host: it is set by a three-parameter transfer $(τ,σ,g_{\min})$. The shading field is divided by its 75th percentile over skin, then gated, scaled, clamped, smoothed, and re-clipped inside a feathered, skin-gated face mask. On an analytic face heightfield, the default $(τ,σ,g_{\min})=(0.90,0.45,0.82)$ modifies 56.5% of face pixels with mean gain 0.938 (0.890 on modified pixels) and drives 3.4% of pixels to the floor. Hue invariance is a corollary of the operator. We analyse the transfer in closed form, ablate its parameters, and discuss failure modes of a monotone, darkening-only formulation, including double-shadowing of non-flat donors.