Generative transformer improves relighting for 3D object images
RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphicsMachine LearningMultimedia
Summary
Adjusting the lighting in photos of objects from different viewpoints is usually tricky and slow. The authors created a new AI model that changes lighting directly on images without needing to figure out complicated 3D details first. They trained this model using a huge dataset of thousands of objects and lighting setups. Their approach works well on images from one or many angles and even for unseen views, producing realistic relighting.
What this means in practice
- •For visual effects artists: Generate realistic lighting changes on digital objects from multiple viewpoints quickly without manual 3D modeling or complex rendering.
- •For e-commerce platform developers: Provide customers interactive views of products under various lighting conditions using fast, learned relighting techniques.$Commercial implications: Enables new product visualization features for online stores by automatically creating photorealistic relighting on diverse object images.
Authors
Hejun Wang, Jinxi Li, Junwei Jiang, Shiwei Mao, Hu Cheng, Shouwang Huang, Bo Yang
Abstract
Image relighting is traditionally tackled via complex inverse rendering pipelines, which suffer from ill-posed optimization, or single-image generative models that ignore crucial multi-view cues necessary for understanding 3D geometry and material interactions. To address these limitations, we introduce a feed-forward generative Transformer for direct single- and multi-view image relighting that entirely bypasses explicit intrinsic property estimation. Adapted from a video foundation model, our architecture features a latent illumination module that dynamically injects target environment maps into spatial features via cross-attention. Furthermore, we employ permutation-invariant positional encodings to symmetrically process unordered multi-view inputs without sequential bias. To train this robust data-driven model, we construct the massive Laval Objaverse Dataset (LOD), comprising 90K objects and 39K unique illuminations. Extensive experiments demonstrate state-of-the-art visual quality, photorealistic relighting quality, and strong zero-shot generalization across single-view, multi-view, and novel-view relighting tasks.