High quality 3D material creation done fully in texture space
Texture Space Material Diffusion
Computer Vision and Pattern RecognitionGraphics
Summary
Creating realistic materials for 3D objects is tricky because materials need to look good from every angle and work on different shapes. The authors developed a method that makes materials directly on the flat 2D surfaces called textures, which wrap around 3D objects. By using this approach with a special AI model trained on video sequences and carefully connecting images to texture spaces, they avoid common problems like inconsistent views. Their method can generate high-resolution materials guided by text or photos, and works well even with complex objects and many pictures.
What this means in practice
- •For 3d game developers: Create detailed and view-consistent materials for game assets entirely from text or photo inputs using their texture space diffusion method.$Commercial implications: Enables production of high-quality materials for games with less manual work, facilitating commercial asset pipelines.
- •For virtual reality content creators: Produce realistic and consistent material textures from multiple photos to improve immersion in VR experiences.
Authors
Jacob Munkberg, Peter Kocsis, Jon Hasselgren
Abstract
We present a method for generating high quality materials for 3D objects entirely in texture space. We finetune a video diffusion transformer for text-guided material generation, multi-view material generation, and material upscaling. Our key insight is to use the known projection from image space to texture space, enabling the diffusion process to generalize across arbitrary geometries and texture parameterizations. This approach also avoids the view consistency issues inherent in video and multi-view diffusion models. Because texture space is two dimensional, we can reuse the strong priors of pretrained video diffusion models. We apply our method to high quality material reconstruction from posed photos captured under unknown lighting, as well as to text- and image guided material generation. Our method can scale to high resolutions (8K), 100+ input views, and neural material representations. In quantitative and qualitative evaluations we show state-of-the-art results for material generation and reconstruction.