Execution logs influence vision language judges on robot progress
Execution Logs Sway VLM Judges of Robot Progress
Machine LearningRobotics
Summary
Judges powered by vision-language models watch robot videos to decide if the robot is making progress. The researchers found that adding a single sentence from the robot’s log, saying it is making progress, can change these judges’ opinions even when the robot is not actually improving. Some models are especially easy to trick with fake claims, while others are more skeptical. Adding a warning that the log could be wrong makes the judges more cautious.
What this means in practice
- •For robotics developers: Improve robot performance assessments by accounting for bias from execution log text when using vision-language models to grade progress.
- •For video content moderators: Detect misleading or manipulated captions in videos when automated systems combine text logs with visual content for judging outcomes.
Authors
Weiyan Zhao, Sunny Yang
Abstract
Vision-language models now grade robot videos, and text placed beside the frames can change their verdicts. On clips where the robot is losing progress, we add one log line saying the robot is making progress. Across Gemini, Qwen and GPT judges, this flips correct verdicts to progress 10 to 35 percentage points more often than a neutral line does. A one-line warning that the log may be wrong removes most of this effect. Selected fake "task complete" claims flip half of Qwen's correct verdicts, against a tenth for unrelated log lines. They have no detectable effect on Gemini.