Code agent grading method improves learning from test results
Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL
Computation and LanguageMachine Learning
Summary
When teaching computer programs to write code using trial and error, we often reward them only for passing tests, giving no extra credit for cleaner or more focused solutions. The authors introduce a new method called GAGAR that compares groups of program attempts and ranks the best solutions according to quality, not just whether they pass tests. This helps the learning process prefer better code and leads to more stable and efficient improvements in code-writing agents. Their experiments with very large code-writing models show this method improves performance and training stability.
What this means in practice
- •For software development teams: Improve automated code generation tools by using quality-ranked rewards rather than binary test results, resulting in cleaner and more reliable code suggestions.
- •For ai model trainers: Enhance reinforcement learning training of large-scale code-writing AI by integrating groupwise grading methods to stabilize training and reduce code bloat.
Authors
Jinhao Dong, Liang Zhao, Zihao Yue, Wenhan Ma, Linghao Zhang, Lei Li, Shicheng Li, Yifan Song, Bowen Ye, Fuli Luo
Abstract
Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and adherence to task requirements. This leaves the policy without a learning signal that favors clean, targeted implementations over those containing unnecessary or out-of-scope changes. We introduce GAGAR, a framework for quality-aware credit redistribution in code agent RL. Built on dynamic sampling that retains groups containing both passing and failing trajectories, GAGAR places all trajectories from each group in a shared workspace, where an SFT-trained agentic grader jointly inspects them and ranks the test-passing candidates. Based on this ranking, we downweight lower-ranked trajectories and proportionally rescale the advantages of all test-passing trajectories to restore their original sum. This sum-preserving redistribution retains the relative weights established by quality-based downweighting while shifting credit toward higher-quality implementations. We evaluate GAGAR at industrial scale using pre-RL SFT checkpoints of MiMo-V2.6-Flash (310B total parameters) and MiMo-V2.6-Pro (1.02T total parameters). Controlled code-only Flash experiments show improved code agent performance, reduced trajectory-length growth, and more stable training. We further apply GAGAR in large-scale mixed-task RL with both Flash and Pro. Our results support combining test-based verification with groupwise agentic grading to improve the quality and stability of code agent RL.