Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models
2026-06-29 • Robotics
RoboticsArtificial Intelligence
AI summaryⓘ
The authors show that Vision-Language-Action (VLA) models can learn and improve their actions without needing external rewards by using their own confidence in decisions as a guide. They introduce T²VLA, a method that lets these models self-improve during testing by comparing their actions to confident expert examples and rewarding similarity. Their approach includes balancing local and global expert knowledge for better learning stability. Experiments demonstrate that T²VLA improves performance on robotic benchmarks and works across different VLA setups without external feedback.
Reinforcement LearningVision-Language-Action ModelsIntrinsic RewardTest-Time LearningTrajectory ConfidencePolicy ImprovementExpert DemonstrationsRobotics BenchmarksSelf-Supervised Learning
Authors
Siyao Chen, Jiakang Yuan, Jiaxin Wang, Tao Chen
Abstract
Reinforcement learning (RL) has become indispensable for pushing Vision-Language-Action Models (VLAs) beyond static imitation learning. However, existing RL methods typically require external environmental feedback, relying on predefined success signals to guide policy updates. In this work, we show that VLA models possess useful internal evaluative capabilities: in discrete-action VLAs, trajectories with higher generation confidence are significantly more likely to succeed. Based on this observation, we introduce T^2VLA (Test-time VLA), an architecture-agnostic test-time RL framework that enables VLA models to achieve self-bootstrapping policy improvement. Instead of relying on external rewards, T^2VLA leverages trajectory-level similarity to high-confidence expert demonstrations as an intrinsic reward signal. In addition, we propose a Confidence-Driven Dual Expert Bootstrapping mechanism, which dynamically balances a Local Pseudo-Expert for exploration and a Global Expert Pool for training stability. Extensive experiments on the LIBERO and RoboTwin benchmarks show that T^2VLA consistently outperforms supervised baselines and approaches oracle RL performance with ground-truth rewards, achieving effective improvement without external reward feedback. Furthermore, T^2VLA adapts to distinct VLA paradigms, including both OpenVLA-OFT and the pi series.