Summary
Vision-language-action systems are usually tested by changing one thing at a time to see how that affects them. This paper shows that when multiple changes happen at the same time, the combined effect can be very different from looking at each change alone. The researchers created a new test to compare single changes with combined changes on the exact same starting situations. They found that sometimes combined changes cause new problems that single changes don’t, and other times combined changes fix problems that single changes cause. This means that testing only one change at a time isn't enough to understand how these systems behave in the real world, where many changes occur together.
What this means in practice
- •For robotics engineers: Evaluate robot vision-language policies under realistic combined changes rather than isolated conditions to better predict real-world failures.
- •For autonomous vehicle teams: Assess combined environmental shifts affecting sensor and language inputs together to improve robustness testing of self-driving cars.
Abstract
Vision-language-action policies are typically evaluated one perturbation at a time, providing a useful diagnosis of their sensitivity to individual distribution shifts. Real-world deployment, however, may involve several shifts simultaneously, and it remains unclear how these individual robustness measurements compose. We ask whether compound robustness can be inferred from single-axis evaluations. We introduce LIBERO-CTRL, a six-axis benchmark that pairs each initial state across single-axis conditions and a matched simultaneous condition. This design reveals two opposing outcome changes that aggregate success rates cannot distinguish: emergent failures, where all single-axis rollouts succeed but the simultaneous rollout fails, and compensated successes, where at least one single-axis rollout fails but the simultaneous rollout succeeds. Because one transition decreases compound success while the other increases it, they can cancel, making aggregate compound performance appear consistent with single-axis measurements even when individual outcomes differ substantially. These opposing transitions can largely cancel in aggregate: even when the difference between the two transition rates is not statistically distinguishable from zero, as many as 29.0% of matched initial states still change outcome. Across six policies and three severity levels, such outcome changes reach 34.5% in the most affected condition. The relative prevalence of the two transitions varies across policies and severities, while the transition rates remain similar under independent re-evaluation of stochastic policies. Compound robustness therefore cannot be characterized from aggregate single-axis success rates alone; matched per-instance evaluation is needed to reveal how joint perturbations alter behavior.