Vision language action models improve robot task success with evolving trust in simulation
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
RoboticsMachine Learning
Summary
Robots and AI systems that learn to act from seeing and language often rely on examples from humans, but these examples don't cover everything a robot might face. The authors propose a new way to help such AI learn better by combining real experience with simulated imagination that evolves its trustworthiness during learning. Their method smartly decides when to trust the simulation and when to rely on real data, improving the robot's success and learning speed on tasks both in simulation and with real robots. This approach helps AI adapt when it encounters new situations beyond its initial training examples.
What this means in practice
- •For robotics engineers: Improve robot control policies by integrating evolving simulator trust to better handle tasks beyond demonstration examples.
- •For autonomous system developers: Enhance learning speed and reliability of AI agents operating in complex environments by balancing real and simulated experiences using trust-calibrated imagination.
Authors
Wenzhan Li, Yiran Qin, Heng Zhou, Huirui Wang, Yulan Guo, Ruimao Zhang
Abstract
Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability--efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.