Policy reward co-evolution struggles to improve code generation generalization
Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift
Computation and LanguageMachine LearningSoftware Engineering
Summary
Generating computer code using trial and error guided by rewards can work well, but the study shows that when trying to improve both the code generator and its reward system together, it doesn't always help with writing new programs. The researchers tested this approach with a specific model and found that even though it got better at the training tasks, it didn't do well on new problems. They also noticed the generation system changed a lot from its original state without actually improving its ability to solve new puzzles. This means simply getting better at the training tasks doesn’t mean the system really learns to write better code overall.
What this means in practice
- •For machine learning engineers: Detect overfitting and policy drift in reinforcement learning models for code generation by comparing training rewards with performance on held-out tasks.
- •For software development teams: Use evaluation techniques from this study to validate whether automatic code generation tools actually improve on new coding problems, not just training examples.
Authors
Amarsaikhan Batjargal, Noriko Tomuro
Abstract
Reinforcement learning with verifiable rewards (RLVR) is particularly well suited to code generation because generated programs can be objectively evaluated through execution. Execution also provides behavioral information that can be reused as additional supervision. The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them. We investigate whether this policy-reward co-evolution approach transfers effectively to code generation and whether execution-grounded reflection provides an additional source of self-improvement. We compare SPARK-style rollout recycling with execution-only Group Relative Policy Optimization (GRPO) using Qwen2.5-Coder-3B-Instruct. Our results do not support either hypothesis under the configuration tested. Although co-evolution improves training performance, these gains do not generalize to held-out problems. Compared at the same stage of training, rollout recycling produces roughly 34x greater KL divergence from the reference policy than execution-only GRPO. Over training, trained tokens per GRPO sequence fall from 185 to 86 as training pass rate increases, while held-out performance declines or fails to sustain early gains. We characterize these patterns as single-adapter policy drift: the shared trainable parameters increasingly depart from pretrained code-generation behavior without corresponding gains in generalization. The results show that higher training reward alone is not evidence of generalizable code self-improvement and highlight the importance of held-out evaluation and drift diagnostics.