LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splatting
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors present LaGSplat, a method that learns how objects move and respond to forces just from one or a few regular videos. Their system uses a small number of important variables (latent states) that describe the object's behavior and link it to a visual representation made of points (Gaussian splats). This setup allows the method to predict how objects, whether stiff or flexible, will react when pushed in new ways not seen before. They tested LaGSplat on real scenarios and showed it can let users interactively apply forces and see realistic reactions in real time. Their approach balances simplicity and realism to avoid unrealistic predictions when forces are unfamiliar.
Latent StateLagrangian MechanicsGaussian SplattingMonocular VideoGeneralized CoordinatesEuler-Lagrange EquationRigid Body DynamicsDeformable ObjectsPhysics-based ModelingInteractive Simulation
Authors
Louen Pottier
Abstract
We present LaGSplat (Latent Lagrangian Gaussian Splatting), a framework that infers interactive, physics-governed dynamics from one or a few monocular videos. At inference it lets a user push on the filmed object, rigid or deformable, with an external force that was never measured, annotated, or seen during training. This is possible because a low-dimensional latent state $\mathbf{q} \in \mathbb{R}^d$ plays two roles at once: it is the generalised coordinate of a learned dissipative Lagrangian and the conditioning variable of a Gaussian Splatting decoder. The inductive bias of this decoder, whose primitives are explicit points $μ_i(\mathbf{q})$ that move with the object, is what lets a force $f$ applied in the image pull back into a latent generalised force $J(\mathbf{q})^\top f$ and enter the equations of motion, which pixel-space (CNN) or neural-field (NeRF) decoders cannot do. We validate LaGSplat on test cases of increasing difficulty, from rigid to deformable and from autonomous to forced real systems, combining monocular video and sensor measurements. We further demonstrate interactive use: forces of arbitrary magnitude and direction can be applied to the reconstructed object at any time, its response rendered in real time, in 2D or 3D. Assuming a dissipative Euler-Lagrange equation over a few generalised coordinates trades generality for a bounded, plausible response to unseen forces, where an unconstrained predictor diverges.