Summary
Making a 3D object from just one photo is very hard because the picture doesn’t show every part of the object. The authors found that using existing AI tools that guess 3D shapes and textures separately leads to issues like wrong colors and inconsistent views. They studied the problem by looking at the details in the data’s frequency components, which are like patterns of different detail levels. Using this understanding, they built a method called Morpheus3D that combines different AI hints in a smarter way to create better 3D objects with correct colors and textures from just one image. Tests on various images show their approach works better than previous methods.
single-view 3D reconstructiondiffusion models2D and 3D priorsfrequency domainhigh-frequency detailsimage-to-3Dtexture reconstructionview consistency
Abstract
Single-view 3D reconstruction, also known as image-to-3D, is a persistently challenging task due to the extreme lack of information. Recently, diffusion models pre-trained on large-scale datasets served as 2D priors are used to solve the ill-posed task but suffer from color deviation and view inconsistency, which can be curbed by using diffusion models fine-tuned with 3D annotated data served as 3D priors. However, 3D priors lack high-frequency details, which cannot be solved by direct complementation with 2D priors in spatial domain for introducing erroneous low-frequency 2D prior guidance. In this paper, we revisit the characteristics of different diffusion priors from the frequency perspective. Based on our observations, we theoretically present a unified framework of hybrid optimization using multiple diffusion priors in frequency domain. Under this framework, we further propose Morpheus3D, a pipeline of 3D object generation from any single unposed image in the wild. Morpheus3D enhances 3D prior with high-pass image-prompt 2D prior guidance to reconstruct high-quality 3D objects while effectively suppressing view inconsistency, low-frequency color deviation, and high-frequency lacking problems. Both quantitative and qualitative experiments on the public and our collected datasets with complex textures show that our method exhibits significant improvements in generation quality.