RDA: Reward Design Agent for Reinforcement Learning
2026-06-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors created a system called Reward Design Agent (RDA) to help robots learn tasks better by understanding instructions more clearly. Unlike previous methods that only looked at simple success or failure signals, RDA uses a vision-language model to check what the robot did, figure out mistakes, and improve the rewards it uses for learning. This approach helps the robot's behavior match the given instructions more closely, while still completing the tasks well. They tested RDA on different robot tasks and found it works better at following instructions than earlier methods.
Reinforcement LearningReward FunctionVision-Language ModelRobotic ManipulationTask AlignmentTrajectory EvaluationPolicy LearningLarge Language ModelReward DesignFailure Mode Analysis
Authors
Hojoon Lee, Ajay Subramanian, Ben Abbatematteo, Vijay Veerabadran, Pedro Matias, Karl Ridgeway, Nitin Kamra
Abstract
Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions. Recent work, such as Eureka, automates reward design by using an LLM to iteratively generate and refine reward code from task descriptions. However, they rely on coarse feedback signals such as success rate, which provide little semantic insight into the learned behavior. As a result, their trained policies achieve the final goal but are frequently poorly aligned with task instructions. We introduce the Reward Design Agent (RDA), a VLM-based agentic framework that injects semantic understanding into reward design. RDA decomposes tasks, visually evaluates trajectories, summarizes failure modes, and iteratively revises reward code to better align with task instructions. Across 12 tabletop manipulation tasks from ManiSkill and 4 whole-body manipulation tasks from HumanoidBench, RDA produces policies substantially more instruction-aligned than those of other baselines, while achieving comparable task success rates. Videos and the generated reward code are available on https://nitinkamra1992.github.io/reward-design-agent.