Real-time adaptation links real deformable object data to simulations
RealSimLoop: Online Real-to-Sim Adaptation via Differentiable Reduced-Order Simulation with Vision Feedback
GraphicsComputer Vision and Pattern RecognitionRobotics
Summary
Simulations can show hidden details about deforming objects, like their stress or forces, but usually don’t match real-world changes quickly. The authors created RealSimLoop, a method that uses camera images to quickly adjust simulation settings to match real objects, even as those objects change over time. It does this efficiently by simplifying the simulation and using image data directly to update the model. Their tests show better results than traditional methods, with uses like predicting forces and visualizing internal stresses from new viewpoints.
What this means in practice
- •For robotics engineers: Adjust simulations of flexible robot parts in real time to improve control and interaction by incorporating live camera feedback.
- •For animation and visual effects teams: Generate realistic internal stress visualizations and new viewpoint renderings of deformable materials for enhanced digital effects.
Authors
Zhihao Cen, Chuhua Xian, Hailin Sun, Yuliang Liufu, Zhen Zhang, Xiangyu Chu, Hongmin Cai, Yunbo Zhang, Guoxin Fang
Abstract
Real-world observations of deformable objects are often sparse or surface-level, while downstream tasks require hidden physical quantities such as internal deformation, stress fields, and interaction forces. Physics-based simulation can recover these quantities, but online real-to-sim adaptation remains challenging due to costly full-space optimization, limited feedback, and time-varying material properties. To address these challenges, we propose RealSimLoop, a differentiable framework for online real-to-sim adaptation using vision data as physical feedback. Our approach achieves quasi-real-time performance by executing differentiable simulation within a reduced-order neural subspace, drastically accelerating the optimization loop. We couple this efficient dynamics model with differentiable rendering, enabling direct gradient backpropagation that leverages high-fidelity pixel data to refine physical parameters such as material stiffness. Furthermore, by employing a sliding-window objective function, RealSimLoop enables robust online adaptation, allowing the system to track time-varying material properties and effectively bridge the real-to-sim gap arising from model reduction or unmodeled dynamics. Extensive experiments demonstrate that our method outperforms conventional offline methods, and we validate the framework's versatility in downstream applications, including external force prediction and 3D stress field reconstruction with novel view synthesis.