Transformer model improves lighting on photos from multiple views
RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphicsMachine LearningMultimedia
Summary
Changing the lighting in photos of objects is hard because it usually needs complicated math or guesses about the object's 3D shape and materials. The researchers designed a new type of computer model, called RelightFormer, that uses a special Transformer to directly change lighting in pictures from one or many angles without figuring out the object's details first. They trained this model on a very large collection of objects and lighting conditions to help it learn well. Their method can create more realistic lighting changes and works well even on new objects or views it has never seen before.
image relightingtransformermulti-view imagescross-attentionpositional encodingintrinsic properties3D geometryenvironment mapszero-shot generalizationLaval Objaverse Dataset
Authors
Hejun Wang, Jinxi Li, Junwei Jiang, Shiwei Mao, Hu Cheng, Shouwang Huang, Bo Yang
Abstract
Image relighting is traditionally tackled via complex inverse rendering pipelines, which suffer from ill-posed optimization, or single-image generative models that ignore crucial multi-view cues necessary for understanding 3D geometry and material interactions. To address these limitations, we introduce a feed-forward generative Transformer for direct single- and multi-view image relighting that entirely bypasses explicit intrinsic property estimation. Adapted from a video foundation model, our architecture features a latent illumination module that dynamically injects target environment maps into spatial features via cross-attention. Furthermore, we employ permutation-invariant positional encodings to symmetrically process unordered multi-view inputs without sequential bias. To train this robust data-driven model, we construct the massive Laval Objaverse Dataset (LOD), comprising 90K objects and 39K unique illuminations. Extensive experiments demonstrate state-of-the-art visual quality, photorealistic relighting quality, and strong zero-shot generalization across single-view, multi-view, and novel-view relighting tasks.