Physically plausible object insertion improves lighting and perspective matching
PhysInsert: Physically Plausible Object Insertion with Geometric Priors and Bidirectional Object-Scene Interactions
Computer Vision and Pattern Recognition
Summary
Inserting objects realistically into images is hard because objects need to match lighting, position, and how they interact with the surrounding scene. The authors created a new dataset using 3D software to capture exact changes from different viewpoints and lighting, helping train models better. They built a system called PhysInsert that uses geometry and image-text guidance to place objects in images more naturally, adjusting their pose and lighting. Tests and user studies showed their method makes inserted objects look more believable and better integrated with the scene.
What this means in practice
- •For graphic designers: Insert objects into images with realistic lighting and perspective adjustments for visual content creation.
- •For augmented reality developers: Enhance virtual object placement in real scenes by improving object pose and lighting consistency with physical surroundings.$Commercial implications: Enables AR apps to place virtual objects realistically in live environments, improving user experience and engagement.
Authors
Yue Gong, Le Zhang, Rui Chen, Song Chang, Xiaochao Qu, Luoqi Liu, Ting Liu, Yebin Liu
Abstract
We present PhysInsert, a framework for physically plausible reference-based object insertion that jointly adapts objects and their surrounding scenes. Existing methods often rely on masked inputs, generation-based pairs, or training pairs with limited viewpoint variation, leaving pose and illumination adaptation and bidirectional object-scene interactions insufficiently constrained. We construct the Insert World dataset with strictly aligned Blender-rendered counterfactual pairs and object references across viewpoints and lighting conditions. Building on these data, Geometry-Guided Adaptation couples a VGGT geometry stream with image and text streams under target-scene feature supervision to guide spatially consistent insertion. Complementary 2D Object-Scene Spatial Supervision jointly supervises object occupancy and environmental changes while grounding spatial instructions in object regions through image-text attention. We introduce PhysInsert-Bench with dedicated cross-environment and cross-view cases requiring illumination and pose adaptation beyond appearance copying. Experiments on PhysInsert-Bench and DreamBooth demonstrate improved image fidelity; a user study further shows more accurate placement and improved lighting integration, perspective, and pose plausibility.