PhysLDM enables detailed simulation of changing 3D soft objects
PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation
Computer Vision and Pattern RecognitionGraphics
Summary
Simulating how soft 3D objects change shape over time is very hard and usually slow or inaccurate. The authors created PhysLDM, a new method that compresses these complex shapes into simpler form so it can predict changes quickly and precisely. They found that using a method called latent diffusion, which generates possible future shapes instead of just one average prediction, works better for unpredictable movements. Their model can predict new situations without extra training and helps solve problems like designing objects by adjusting shapes automatically.
What this means in practice
- •For graphics developers: Simulate realistic soft body deformations for visual effects and animations with high detail and temporal accuracy.
- •For industrial designers: Use PhysLDM’s differentiable simulation for efficient shape design adjustments and optimization in product development.
Authors
Yu Zhang, Xudong Xu, Xingang Pan
Abstract
Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. Long-horizon prediction for high-resolution 3D volumetric meshes is hard: autoregressive methods are susceptible to error accumulation, while direct multi-frame prediction at native resolution is computationally prohibitive. This motivates a compact spatiotemporal latent representation, which is largely unexplored for mesh-based volumetric physics. Meanwhile, it remains unclear whether deterministic regression or generative diffusion is the more appropriate predictive paradigm. To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot volumetric deformable simulation. Its core is a holistic spatiotemporal VAE that avoids the "staircase" artifacts of standard temporal compression (as in common video VAEs), achieving ~2.48 mm reconstruction precision on meter-scale scenes at up to 78x token compression. Based on this reliable latent space, we systematically compare regression and diffusion methods. Our experiments uncover a key modeling insight: complex deformable dynamics are often chaotic, and in this regime deterministic regression tends to produce non-physical averages, whereas diffusion better models their distribution. Accordingly, we employ a latent diffusion model that effectively learns from the chaotic data to generate physically plausible trajectories. Trained purely kinematically on an Objaverse-scale dataset, a single PhysLDM generalizes zero-shot to unseen OOD datasets (GSO and Toys4K). Its differentiability further enables efficient solution of inverse problems and higher-order design optimization. To our knowledge, PhysLDM is the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation.