Robot learns to predict physics by interacting with environment
Closing the Dynamics Gap through Self-Directed Robotic Interaction
RoboticsArtificial Intelligence
Summary
Robots can create digital replicas of scenes they see, but they struggle to understand tricky physical behaviors like friction without touching things. The authors introduce CUE, a robot system that learns these physical details by playing and experimenting on its own. With only simple instructions and no resets, CUE figures out the hidden physics by focusing on where its predictions are least accurate. This approach greatly improves its ability to predict how objects like billiard balls move, both in simulations and on a real pool table.
What this means in practice
- •For robotics engineers: Improve robot interaction skills by enabling robots to self-learn physical parameters through autonomous environment probing without manual resets.
- •For industrial automation teams: Enhance robotic handling accuracy in contact-rich tasks by refining simulations with autonomous robot-driven data collection.
Authors
Xilun Zhang, Liam Achenbach, Luis Pabon, Matthew Foutter, Marco Pavone
Abstract
Agentic Real2Sim2Real systems can reconstruct a scene in simulation from camera observations, but contact-rich dynamics such as friction and restitution cannot be observed without interaction. We call this the dynamics gap and present CUE, an agentic system that closes it by interacting with the environment itself. Given only a high-level instruction and no resets, the agent builds its own perception, identifies physical parameters, and targets interactions where its simulation is least accurate. On billiards, CUE reduces held-out ball-position error by 72% relative to the strongest baseline across 25 hidden-physics conditions in simulation, and we further deploy it on a real pool table. More details can be found on the project website: https://cue-dynamics.github.io/