Generative models judged on how well they match photo lighting
Shedding Light: A Benchmark for Evaluating Lighting Understanding in Generative Image Models
Computer Vision and Pattern Recognition
Summary
Realistic pictures depend a lot on correct lighting, but it’s unclear if AI models that create images really understand lighting properly. The authors developed a test to see how well these models can insert objects into photos while matching the scene’s light correctly. They used special photos with objects that show the light’s direction and color, then compared the AI’s added objects to the real ones. This lets them measure how accurately the AI models understand and reproduce real-world lighting.
What this means in practice
- •For digital artists and designers: Assess how well generative tools blend new elements with realistic lighting in photographs, improving image editing workflows.
- •For augmented reality developers: Evaluate and improve lighting consistency when inserting virtual objects into real-world scenes for believable AR experiences.
Authors
Justine Giroux, Jack Oliver Hilliard, Yannick Hold-Geoffroy, Javier Vazquez-Corral, Jean-François Lalonde
Abstract
Accurate modelling of illumination is central to realistic image synthesis and scene understanding. Yet, there is little exploration into whether image generative models are good at this task or whether physical plausibility remains a key challenge for them. Clearly, significant progress has been made in realistic image synthesis, but do models truly understand lighting in a physically accurate manner? To answer this question, this work proposes a benchmark to assess the lighting understanding and harmonisation capabilities of generative models. Our key insight is that evaluating lighting understanding for such models only requires testing how well they insert novel objects into real photographs whilst maintaining consistent illumination. To do so, we use a multi-illumination dataset with images containing simple objects serving as ``light probes'', and prompt models to inpaint the same object onto the original image, then compare the generated results against the ground-truth light probes. We then estimate the lighting direction, colour and radiance distribution from the inpainted probes, providing a quantitative measure of illumination accuracy and photometric realism. Our work establishes a scalable evaluation protocol to systematically assess how well generative models capture and reproduce real-world lighting, offering a foundation for benchmarking the photometric accuracy of any future models. All code and data are available at https://lvsn.github.io/SheddingLight/ .