UltraTex improves 3D texture maps with physical and material knowledge

UltraTex: Aligning Physical Knowledge for High-Fidelity PBR Texture Generation

Computer Vision and Pattern Recognition

Summary

Creating realistic surface textures for 3D objects from a single photo is difficult because the photo mixes up material, lighting, and shape. The authors introduce UltraTex, a method that uses multiple views and physical knowledge about materials to make better texture maps. UltraTex predicts color and shape details first, then estimates metal and roughness properties by understanding the type of material and how light interacts with it. This approach helps produce more accurate and realistic textures than existing techniques.

What this means in practice

  • For 3d artists and animators: Generate more accurate PBR texture maps from single images to achieve realistic material appearance for 3D assets used in films and games.
  • For augmented reality developers: Create physically plausible material textures from photos to improve realistic lighting and reflections in AR applications without extensive manual tuning.

Authors

Jingzhi Bao, Xiaohe Ma, Hong Li, Yuanming Hu, Xiaoguang Han

Abstract

Generating physically based rendering (PBR) texture maps for a 3D mesh from a single reference image is highly ambiguous, as the observed appearance entangles material identity, surface reflectance, and illumination. While current generative methods produce realistic albedo maps, they still struggle to infer accurate metallic and roughness attributes for relighting. We present UltraTex, a cascaded multi-view framework that moves beyond appearance matching by integrating physical knowledge into multi-view diffusion-based PBR generation. UltraTex first jointly predicts albedo and world-space shading normals to generate more detailed albedo maps, and then predicts metallic-roughness (MR) maps conditioned on the generated albedo, reference image, and material semantic priors. To inject material semantics, a lightweight adapter transforms hidden-layer features of a vision-language model (VLM) into intrinsic material representation tokens, providing part-level knowledge anchors for MR estimation. To model material-light interaction, photometric representation alignment guides intermediate diffusion features toward ground-truth material renderings under controlled lighting. This makes metallic and roughness variations observable through highlight intensity, specular sharpness, and local reflectance contrast, providing stronger guidance for fine-grained material prediction without explicit lighting supervision. Extensive experiments show that UltraTex generates high-quality textures and outperforms existing open-source and commercial methods in visual fidelity, material accuracy, and physical plausibility.