Human-centered video relighting enables precise light control using joint rgb and irradiance modeling
JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling
Computer Vision and Pattern Recognition
Summary
Changing the lighting in videos with people is challenging because lighting is complex and hard to control. The authors create a new method that combines color images and light information to improve lighting adjustments. They train their method on detailed synthetic videos and show it works well on real videos, giving better light effects and control than before. This approach helps make video lighting look more natural and accurate.
What this means in practice
- •For video production teams: Enable precise and flexible lighting adjustments in human-centered videos for improved visual quality during post-processing.
- •For game developers: Enhance character lighting realism in real-time animations by integrating controllable lighting effects based on learned irradiance cues.$Commercial implications: Allows sale of advanced lighting modules for interactive character rendering with high fidelity and flexible control.
Authors
Henglei Lv, Bailin Deng, Xiaoqiang Liu, Yichen Wang, Haoxian Zhang, Pengfei Wan, Lin Gao
Abstract
Understanding and modeling light transport in human-centric scenes is a longstanding challenge. While recent progress in diffusion models has boosted research in relighting, precise illumination control still lags, primarily due to the scarcity of diverse, high-quality, precisely labeled video relighting datasets. Existing end-to-end relighting approaches model light transport implicitly through neural networks, which substantially complicates controllability. In contrast, some works resort to path-traced irradiance cues as an explicit blueprint for light transport. However, such constructed irradiance cues are prone to artifacts and hinder generative quality when directly used as control signals. With the key observation that noise corruption removes high-frequency artifacts and preserves low-frequency light transport, we introduce an asynchronous joint-modeling paradigm that facilitates fine-grained illumination control. During two-stage training, we force the model to jointly denoise RGB and ground-truth irradiance frames with separate timesteps scheduling. A domain matching strategy is applied to the second stage to bridge the domain gap between irradiance cues and ground-truth irradiance maps. At inference time, we introduce an asynchronous hijacked generation pipeline that facilitates flexible illumination control. For training data, we develop a multi-illumination high-quality synthetic video dataset with paired irradiance maps using a powerful game engine. Trained exclusively on this dataset, our model exhibits robust generalization to real- world scenarios and outperforms prior approaches in both lighting fidelity and generative quality. The code and dataset will be publicly released.