Relightable 3d scene stylization keeps lighting effects separate from colors
RxGS: Relightable Gaussian Stylization via BRDF Decomposition
Computer Vision and Pattern Recognition
Summary
Stylizing 3D scenes with reference images usually mixes up the color of surfaces with lighting effects, making it impossible to change the lighting later. The authors created a method called RxGS that separates the surface color (albedo) from lighting and material properties. This allows you to change the look of a scene while still being able to relight it under different conditions. Their method performs well on test scenes and offers a way to stylize scenes that remains physically accurate for new lighting.
What this means in practice
- •For 3d graphics developers: Create stylized 3D scenes that can still be realistically relit under varying lighting conditions.
- •For visual effects teams: Produce 3D scene stylizations that preserve physical materials and lighting for consistent rendering in different environments.
Authors
Wanrong Hong, Liu Jiang, Mukesh Prasad, Jun Li
Abstract
Reference-based 3D scene stylization transfers the appearance of a reference image onto a reconstructed 3D scene while preserving scene geometry. Recent methods perform this transfer by optimizing the appearance representation of neural scene models including Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). These methods optimize an RGB appearance representation, in which a single per-point colour combines the effects of material reflectance, environment illumination, together with the shading it produces, and the viewing direction. When stylization modifies this combined colour, the illumination effects present during stylization are baked into the stored appearance, so the stylized scene cannot be rendered under different illumination. To address this limitation, we present RxGS, a relightable 3D scene stylization framework built on a Bidirectional Reflectance Distribution Function (BRDF)-aware 3D Gaussian representation. Rather than modifying a per-point RGB colour, RxGS optimizes only the diffuse albedo field recovered by inverse rendering, while leaving roughness, metallicity, surface normals, and the estimated environment lighting unchanged. As a result, the stylized scene keeps all the components a physically-based renderer needs other than the edited albedo, and can therefore still be relit. RxGS optimizes the albedo field using a nearest-neighbour feature matching loss together with a spatial smoothness prior over Gaussian centres. We evaluate the method using a Geometry Consistency Cost (GCC) based on optical-flow warping error. RxGS achieves the lowest GCC on all evaluated NeRF-Synthetic scenes and remains competitive on real-captured scenes, while additionally supporting relightable stylization, which existing reference-based 3D scene stylization methods do not provide.