Robot uses vision and language skills to judge and improve its tasks
Intrinsic Robot Rewarding: Reusing VLA Representations for Autonomous Evaluation and Policy Improvement
RoboticsMachine Learning
Summary
Robots need to know if they did a job well to get better at it. The authors suggest using the robot’s own eye and language understanding system, which already helps it see and follow instructions, to also judge how well it did a task. This means the robot can check its work without needing extra tools or people to tell it if it was right. They show how this idea works with a real robot arm, making it easier and faster for robots to learn from experience.
What this means in practice
- •For industrial robot programmers: Automatically evaluate robot task outcomes using existing visual-language features to improve performance without extra evaluation tools.
- •For warehouse automation teams: Implement in robot fleets to reduce the need for human outcome checks by reusing vision-language components for self-assessment.
Authors
Tobias Schaffer, Mohab Elkhayat, Daniela Nicklas, Mustafa Almohamad, Elham Al-Fuqara
Abstract
Vision-language-action (VLA) systems already bring together two valuable resources for robot learning: rich visual representations and demonstrations of successful task execution. Intrinsic Robot Rewarding (IRR) proposes to use these resources for a second, complementary purpose: evaluating the robot's own outcomes and providing feedback for policy improvement. Successful demonstration endpoints define task-specific references, and the policy's frozen visual encoder provides the feature space in which new outcomes are assessed. The core reward mechanism adds a reference bank and a scoring operation to the existing pipeline, without requiring a separate learned evaluator or an additional perception backbone. Our position is that this reuse offers a promising route to lower integration effort, efficient reward computation, and reduced recurring human outcome scoring. Building on established research in visual rewards and learning from experience, IRR brings these ideas into the robot's existing perception and demonstration pipeline. An operational COMAU Racer 3 demonstrator is available at technology readiness level 4 (TRL 4). This laboratory foundation supports the next research step: connecting internal outcome evaluation to physical policy improvement. We present the reward formulation, central research questions, and an evaluation methodology linking reward reliability to task success and supervision effort. The intended contribution is a reusable approach to learn and improve from the data and experience already available in industrial robot systems.