Summary
Capturing both how objects look and how they behave when touched or deformed is tricky. The authors present a new method that first reconstructs an object’s detailed 3D shape and appearance from images, then models how that object physically deforms using a physics-based method called smoothed particle hydrodynamics (SPH). Their approach connects the visual model and physical simulation in a way that can be differentiated, meaning it can be used to figure out material properties or boundary conditions from observed deformations. They tested their method on examples like beams and shapes resembling organs, showing it can simulate and also estimate physical parameters even with noisy data.
What this means in practice
- •For medical simulation developers: Simulate and analyze deformation of organ-shaped models from visual data for surgical planning and training.
- •For mechanical design engineers: Estimate material properties and mechanical parameters of physical prototypes from visual deformation observations during testing.
Abstract
Recent advances in generative world models have increased interest in digital models that reproduce both the appearance of real objects and their response to physical interaction. Three-dimensional reconstruction techniques, including 3D Gaussian Splatting, capture detailed surface geometry and appearance from images and videos. However, extending these representations beyond plausible animation to mechanically interpretable models for constitutive behavior, boundary conditions, and inverse parameter identification remains less explored. In this work, a differentiable Lagrangian-coupled 3DGS-smoothed particle hydrodynamics (SPH) model is proposed for forward simulation and inverse analysis of deformable solids. The observed object is first reconstructed from multi-view calibrated visual dataset as a 3DGS rendering model. An envelope-based procedure then generates an independent SPH support for the solid-mechanics model, avoiding the direct use of rendering primitives as mechanical particles. A reference-configuration Lagrangian transfer maps SPH deformation to Gaussian positions and covariances, thereby coupling the physical model and the image observation model while preserving a differentiable computational path. The SPH formulation supports linear elastic, hyperelastic, and Kelvin--Voigt viscoelastic responses, together with fixed, free, and Robin-type boundary conditions. Numerical studies validate the SPH response against finite-element results, assess accuracy and efficiency against a conventional model using Gaussian centers as surface SPH particles, and demonstrate forward simulations on beam, bridge, and liver-shaped examples. Inverse analyses further estimate constitutive and boundary parameters from rendered deformation observations, including noisy cases, demonstrating the feasibility of the proposed model for mechanics-based parameter identification from image data.