AI summaryⓘ
The authors created Act2Answer, a way to test if vision-language-action (VLA) models—robots that see and understand language—still remember important facts after being trained on robot tasks. Instead of just answering questions verbally, these models show their answer by moving and placing objects, which makes it easier to see if they really know the answer or just guess. They tested several models and found that while these robot models do well on simple ideas, they struggle more with deeper knowledge compared to their original language-vision models. The authors also found that training with question-answering tasks helps robots keep more knowledge, and interestingly, important information is found mostly in the middle layers of the models.
Embodied Vision-Language-Action (VLA) modelsVision-Language Models (VLMs)Commonsense KnowledgeFactual KnowledgeRobotics DataAct2AnswerAction-grounded EvaluationLayerwise Intent ProbingVisual Question Answering (VQA)Model Layer Analysis
Authors
Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro
Abstract
Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action. Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and world-knowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. Act2Answer is available at https://tttonyalpha.github.io/act2answer/.