Diffusion transformers model dynamic 3d clothing on moving bodies
DiT-Garment: Garment Dynamics with Diffusion Transformers
Computer Vision and Pattern Recognition
Summary
Modeling how clothes move on people who are walking or posing is tricky because clothes can have many shapes and materials. The authors created DiT-Garment, a method that uses a kind of AI called diffusion transformers to predict how any clothing design moves with a moving body. Their method works even for clothes it has never seen before, by learning the shape changes in a special 2D space related to the clothing surface. This helps in making realistic animations of clothes in different poses and materials without complicated setup for each new garment.
What this means in practice
- •For computer graphics developers: Generate realistic animations of various clothing designs on digital human models in different poses without custom templates.
- •For virtual try-on platforms: Simulate how new garment designs will fit and move on users’ digital avatars to improve online clothing visualization.$Commercial implications: Enables improved virtual fitting experiences for apparel retailers by realistically modeling garment dynamics on avatars.
Authors
Antoine Dumoulin, Laurence Boissieux, Joao Regateiro, Pierre Hellier, Stefanie Wuhrer
Abstract
We present DiT-Garment to model dynamic 3D clothing over human body models in arbitrary motion. Unlike existing methods, DiT-Garment can animate garments with unseen designs and physical materials, while allowing for direct inference of deformations for any target pose. To achieve this, we leverage a 2D diffusion transformer architecture to learn 3D deformations in a 2D UV-space. As the result is non-deterministic, our generative model learns the distribution of possible outcomes. The template garment is represented as a 3D triangle mesh spatially aligned with a 3D human body model in a standardized pose. To work with different garment designs without the need of a common template or complex graph convolution operations, the diffusion transformer is conditioned on a 3D position map of the template, represented in UV-space, which allows to implicitly learn a deformation of the 3D space around the body in standard pose. Further conditioning on body motion and physical parameters allows to physically ground the model. We quantitatively and qualitatively evaluate DiT-Garment on both synthetic and real data. While only trained on synthetic simulations of automatically generated cloth designs, our method generalizes to captured and artist-made garment designs. Code and data are available for research purposes at https://dumoulina.github.io/dit-garment/.