Dynamic texture creation adds changing looks to 3d objects
DynaMesh: Dynamic 3D Texture Generation
Computer Vision and Pattern Recognition
Summary
Creating changing visual effects on 3D objects is hard because existing methods either change the shape or just paint fixed textures. The authors developed DynaMesh, which makes textures that evolve over time on a fixed 3D shape based on text descriptions. Their method avoids flickering or shape changes by blending details over time and keeping the shape constant, producing a single 3D mesh with textures for each moment. This approach works well on different objects and effects, even those not seen before during training.
What this means in practice
- •For 3d artists and animators: Generate evolving visual effects on 3D objects without altering their shape to create more dynamic and realistic animations.
- •For game developers: Create textured 3D characters or objects with time-varying appearances that enhance immersion and visual storytelling in games.
Authors
Raj Hansini, Guan Chen, Rana Hanocka, Itai Lang
Abstract
We present DynaMesh, a dynamic texture generation method for 3D meshes. Given a textureless shape and a text prompt describing an effect, our method produces an appearance that evolves while the object's geometry remains unchanged. Previous works on dynamic 3D content generation have focused on motion, where an object's geometry and position change while keeping its appearance the same. Methods on texture generation sit on the other side of the problem, painting appearance onto a shape as a fixed surface property and not as an evolving process. Neither addresses a visual effect that propagates on a 3D object. A natural route consists of two generators: a video model that shows the effect from a single view, and an image-to-3D generator that lifts each frame to 3D. However, the latter has no notion of time, so running it per video frame produces a sequence that flickers, loses effect details, and yields a different mesh at every video frame. Our method addresses these failures by conditioning a video model on a render of the mesh and the prompt to obtain a reference video, then running a frozen image-to-3D generator on the video with two changes. The conditioning of each frame is blended over a temporal window, and low-rank adapters are fit per shape to restore the lost details. The mesh is encoded once for the whole sequence, so geometry is constant by construction, and the output is a single mesh with a texture per frame. Applied to various objects and effects, DynaMesh substantially improves over recent video-to-4D and texturing methods, and can generalize its temporal effect to different shapes never seen during training. Our project page is at https://threedle.github.io/dynamesh/.