Reflection aware method improves 3D view generation with mirrors
Reflection-aware Generative Novel View Synthesis
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating images of a scene from new viewpoints can be tricky when mirrors are involved because reflections show parts of the scene from different angles. The researchers developed a method called Ref-GeNVS that cleverly treats reflections as extra camera views without needing to retrain models. Their approach estimates where mirrors are and uses that to create new virtual viewpoints, improving the consistency of reflections in generated images. This helps produce realistic new views that include details visible only through mirrors in both synthetic and real-world scenes.
novel view synthesismulti-view diffusion modelmirror reflectioncamera pose estimationvirtual viewpointreflection consistencygenerative modelsmirror planeimage synthesisreflection injection
Authors
GeonU Kim, Shin Dong-Yeon, Tae-Hyun Oh
Abstract
We propose Ref-GeNVS, a training-free, reflection-aware method for generative novel view synthesis (NVS) in mirror scenes. Existing multi-view diffusion models often fail to recognize the mirror in the scene and cannot exploit reflected content for scene generation. To fix this issue without additional training, our key idea is to treat a mirror image as two complementary views. From input images, we estimate the mirror plane and reflect camera poses to form virtual views. Based on this virtual view setup, we propose a two-stage generation method consisting of Mirror-gated attention and Reflection injection, which enables reflection-consistent NVS by explicitly leveraging reflection relationships in a multi-view diffusion model. Ref-GeNVS inherits the strong generalizability of the multi-view diffusion backbone, while it does not require finetuning. On synthetic and real scenes including mirrors, Ref-GeNVS outperforms recent generative NVS methods by generating reflection-consistent and contextually coherent novel views, revealing scene structure visible only through mirrors. Project page: https://kim-geonu.github.io/Ref-GeNVS/