Video Generation Models Are Inherent Lighting Estimators
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors propose V-LITE, a method to extract realistic, changing lighting information from regular videos. They do this by adding a fake shiny ball into video frames to help the model learn how light bounces in the scene. They improve video generation models to handle high dynamic range (HDR) lighting and use special training techniques to make the results more accurate. Their experiments show that these video models can estimate lighting realistically over time, not just create images. This helps in making computer-generated scenes look more natural under complex lighting.
dynamic environment mapsvideo generation modelsvideo inpaintingchrome ball reflectionspatio-temporal contextHDR (high dynamic range)VAE (variational autoencoder)LoRA fine-tuningdiffusion modelsphotorealistic rendering
Authors
Ziqi Cai, Shuchen Weng, Kaiqi Liu, Zifeng Wang, Zhiquan Zhang, Minggui Teng, Han Jiang, Boxin Shi
Abstract
Recovering dynamic environment maps from a single in-the-wild video is crucial for photorealistic rendering, yet remains a challenge. Recent video generation models can produce photorealistic scenes with complex lighting, possessing an inherent understanding of lighting. In this paper, we introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing lighting estimation as a guided video inpainting task. Inspired by VFX industry practices, we insert a synthetic chrome ball into the scene to compel the model to generate physically plausible reflections from the surrounding spatio-temporal context. To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy. We then construct a mixed dataset comprising high-fidelity HDR images to provide realistic HDR priors, and in-the-wild HDR videos to provide dynamic spatio-temporal context. Extensive experiments demonstrate that V-LITE produces temporally coherent HDR environment maps, revealing that modern video diffusion models are not merely synthesizers but also powerful, inherently capable estimators of physical scene lighting.