Robot world model dynamics evaluated with focus on uncertainty accuracy
Right Ranking, Wrong Scale: Reproducing the Robotic World Model's Dynamics Model and Testing the Uncertainty of Its Follow-up
RoboticsMachine Learning
Summary
Predicting how robots will move is important for teaching them to act safely and well. The authors recreated a robot movement prediction model called RWM and tested how well its uncertainty guesses match actual errors. They found that while the ranking of uncertain predictions aligns well with mistakes, the model usually underestimates how wrong it might be. The study also fixed some issues in the original evaluation and tried ways to better reflect uncertainty over longer times.
What this means in practice
- •For robotics engineers: Improve offline policy optimization by better quantifying uncertainty in robot motion predictions to avoid overconfident errors.
- •For control systems developers: Use ranked ensemble disagreement to identify when predicted robot trajectories are most likely to be inaccurate for safer control design.
Tested on one dataset.
Authors
Joyjeet Singh
Abstract
We reimplement the dynamics model of the Robotic World Model (RWM) and test the uncertainty estimate that its follow-up, RWM-U, uses to penalize reward in offline policy optimization. Our implementation matches the released model's gradients. On held-out data, training on the model's own rollouts beats one-step training at the preregistered long horizon for all ten episodes, but loses on short one-step windows. RWM-U's ensemble disagreement ranks rollout errors better than forecast step (Spearman rho = 0.90 at 100 steps), yet its spread substantially understates error: one-sigma coverage on the released checkpoint is 4.6% at 100 steps, versus 68% for a calibrated Gaussian. Rolling ensemble members on their own predictions raises coverage to 33%. The implemented squared-error loss on sampled predictions drives each member's variance toward zero. A horizon-specific rescaling restores one-sigma coverage under leave-one-episode-out evaluation, but not two-sigma coverage. We also identify two defects in the released evaluation pipeline. The study covers one robot, one gait and one dataset; it trains no policy.