Physically based rendering techniques directly shape image generation features
Physically Based Rendering in the Latent Space
GraphicsArtificial IntelligenceMachine Learning
Summary
Creating realistic images on computers often balances two worlds: classical methods that simulate light in the real world, and newer AI methods that generate images from learned patterns but are harder to control. The authors found a connection between how light behaves and the hidden features inside AI models that create images. By combining physics-based light simulation with the AI’s internal features, they can guide the generation process more predictably. This allows them to adjust scene settings like lighting and viewpoint and produce images that match those changes with minimal extra work.
What this means in practice
- •For visual effects artists: Adjust scene lighting and geometry interactively while generating images consistent with physical light transport, reducing manual tuning.
- •For game developers: Generate game environment visuals with physically plausible lighting and flexible scene changes guided by latent space simulations.
Authors
Vuk Radovanovic, Vishesh Gupta, Adrien Gruson, Binh-Son Hua
Abstract
Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.