Vision language action models show varied results across household tasks

What Actually Matters? A Paired Evaluation of Inference Settings for $π_{0.5}$ on BEHAVIOR-1K

RoboticsArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning

Summary

Vision-language-action models aim to help robots perform many different household tasks by understanding images and language together. The authors studied how well different ways of using these models work across 100 tasks. They found that methods that do well in small experiments don't always work well when scaled to many and varied tasks. The findings come from testing a model tuned for a major robotics challenge, showing that there’s no one-size-fits-all approach yet.

What this means in practice

  • For robotics engineers: Improve robotic system design by selecting inference methods tailored to specific household tasks from diverse environments.
  • For automation software developers: Develop better data collection pipelines that match task diversity to improve robotic task performance in complex, real-world settings.

Authors

Ruben Hammele, Samira Huber, Mohammad Nazeri, Amirreza Payandeh, Alexandyr T. Card, Aaron Nguyen, Anuj Pokhrel, Daeun Song, Xuesu Xiao, Sören Pirk

Abstract

Vision-Language-Action (VLA) models offer a promising paradigm for general-purpose household robotics. However, inference and data-collection strategies that have been deployed in smaller-scale settings do not necessarily generalize to large-scale, diverse task suites. To address this gap, we conduct a systematic, paired evaluation of previously proposed inference and data-collection strategies across all 100 tasks. Our evaluation uses a fine-tuned $π_{0.5}$ model based on the 2025 BEHAVIOR Challenge winner, which achieves a Q-score of 0.39 on the public test set. These results show that proposed strategies do not transfer uniformly to new environments and tasks. Project page, videos, and detailed analysis: https://rubenhammele.github.io/behavior2026