Alignment method improves realism in 3D asset generation
OREO: Fidelity Alignment in 3D Generation via On-the-fly Rendering-Editing Optimization
Computer Vision and Pattern RecognitionGraphics
Summary
Creating realistic 3D objects on computers is challenging because many models produce visuals that don’t look quite right. The authors introduce a method named OREO that uses a loop where 3D objects are rendered into images and then improved using 2D image tools. These improved images guide the 3D model to make better-looking objects over time. This approach helps the 3D generator learn from its own mistakes and gradually create more lifelike visuals.
What this means in practice
- •For 3d asset creators: Enhance the visual quality of 3D models by integrating on-the-fly edited images as feedback in the generation process.
- •For game developers: Produce more realistic 3D game assets by leveraging continuous refinement from 2D image feedback within the generation pipeline.
Authors
Zhiyuan Ma, Wenbo Hu, Wang Zhao, Pengfei Wang, Ying Shan, Lei Zhang
Abstract
Despite recent advancements in 3D generation, models often struggle to produce assets with high visual fidelity. To bridge this gap, we propose OREO, an alignment framework that enhances the realism of 3D generators by leveraging rich 2D diffusion priors. Instead of relying on static datasets, OREO establishes a dynamic optimization loop that produces on-the-fly edited renderings as 2D pseudo-targets. At its core, we introduce Reinforced Editing, which utilizes a 2D model to refine rendered views of the 3D output, enhancing their overall visual fidelity while preserving the underlying geometry, viewpoint, and content. These refined views serve as high-quality supervision targets, enabling the 3D generator to learn from its own generated samples and progressively improve its visual quality. Experiments demonstrate that OREO effectively improves upon pre-trained baselines, producing 3D assets with enhanced visual realism.