Image based 3D objects generated more accurately with frequency based method
Single Image to Textured 3D Object Generation in Frequency Domain: From Theory to Pipeline
Computer Vision and Pattern Recognition
Summary
Turning a single photo into a detailed 3D model is very hard because one image has limited information. The authors looked at how existing AI tools handle details and colors differently in various parts of the image's frequency spectrum. They designed a new approach that mixes 2D and 3D AI models more cleverly by focusing on different frequency parts, which helps create 3D objects with better details and consistent colors from just one unposed image. Their new pipeline, Morpheus3D, shows better results on various textured objects than earlier methods.
What this means in practice
- •For game developers: Generate high-quality 3D game assets from a single concept image with better texture and color consistency.$Commercial implications: Enables automated creation of detailed textured 3D models for games from single images, saving development time and costs.
- •For augmented reality designers: Create realistic 3D objects from a single photo for AR apps with improved detail and fewer visual artifacts.
Authors
Qisen Wang, Yifan Zhao, Jia Li
Abstract
Single-view 3D reconstruction, also known as image-to-3D, is a persistently challenging task due to the extreme lack of information. Recently, diffusion models pre-trained on large-scale datasets served as 2D priors are used to solve the ill-posed task but suffer from color deviation and view inconsistency, which can be curbed by using diffusion models fine-tuned with 3D annotated data served as 3D priors. However, 3D priors lack high-frequency details, which cannot be solved by direct complementation with 2D priors in spatial domain for introducing erroneous low-frequency 2D prior guidance. In this paper, we revisit the characteristics of different diffusion priors from the frequency perspective. Based on our observations, we theoretically present a unified framework of hybrid optimization using multiple diffusion priors in frequency domain. Under this framework, we further propose Morpheus3D, a pipeline of 3D object generation from any single unposed image in the wild. Morpheus3D enhances 3D prior with high-pass image-prompt 2D prior guidance to reconstruct high-quality 3D objects while effectively suppressing view inconsistency, low-frequency color deviation, and high-frequency lacking problems. Both quantitative and qualitative experiments on the public and our collected datasets with complex textures show that our method exhibits significant improvements in generation quality.