Vision language action models show varied robot behaviour despite task success
Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks
Robotics
Summary
The paper looks at how robots controlled by Vision-Language-Action (VLA) models perform tasks despite changes or disturbances during execution. While these models often succeed at completing tasks, the authors found that the way robots behave during successful attempts can change a lot when conditions are altered. They argue that just measuring if a task is completed isn’t enough; we should also look at how the robot carries out the task to better understand its robustness. The authors provide tools to evaluate this behaviour using motion and gripper movement metrics.
What this means in practice
- •For robotics developers: Improve robot manipulation systems by evaluating not just task success but also behaviour consistency under disturbances.
- •For industrial automation engineers: Enhance reliability assessments of robotic arms in manufacturing by using additional metrics that capture execution quality, not just completion.
Authors
Sophie Higham, Riccardo Andrea Izzo, Matteo Matteucci, Alessandro Suglia
Abstract
Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphasis on evaluating the robustness of VLA models to perturbations. However, this robustness is still predominantly measured through Task Success Rate (TSR). In this work, we propose a benchmark-agnostic evaluation framework to measure the behavioural robustness of models by characterising how successful trajectories are executed under perturbation. We implement this methodology by extending the widely-used LIBERO and LIBERO-Plus benchmarks. Across three state-of-the-art VLA models, four LIBERO task suites and seven perturbation conditions, we evaluate changes in both typical successful behaviour and its variability, including metrics of motion smoothness, efficiency and gripper behaviour. We find that perturbations can alter the behaviour of successful trajectories, a phenomenon which cannot necessarily be inferred from TSR alone. Across LIBERO suites, we identify cases where state-of-the-art VLA models achieve comparable TSR under the same perturbation condition, yet behaviour on successful trajectories diverges substantially. Therefore, to have a more robust assessment of task performance, we argue that suitable measures of robustness should capture not only whether a task is completed, but also how the robot behaves while completing it. When evaluating the robustness of VLA models, TSR may be complemented by behavioural evaluation metrics that characterise the nature and variability of successful task execution by robots.