Early reward predictions speed up coding agents handling long coding tasks
Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agents
Computation and Language
Summary
Long coding tasks are tough for AI agents because they only get feedback at the very end, which makes learning slow and unstable. The authors introduce a new method called Contextual Early Reward (CER) that predicts how well the agent will do before it finishes the entire task. By learning from similar past tasks, CER helps the agent avoid wasting time on bad guesses and speeds up decisions. Tests show CER can match or improve performance while using fewer computing resources.
What this means in practice
- •For software engineering teams: Improve automated code generation systems by providing earlier and more reliable reward signals during long coding tasks.
- •For machine learning engineers: Reduce computational costs and training time in reinforcement learning for agents that write or repair code by using early reward predictions.
Authors
Jihan Yao, Sihan Zeng, Shangbin Feng, Zhiyuan Fan, Banghua Zhu, Yulia Tsvetkov
Abstract
Long-horizon coding agents receive verifiable rewards only after completing expensive sequences of tool calls. This increases inference cost, amplifies early wrong hypotheses, and can lead to sparse terminal reward and unstable training. We introduce Contextual Early Reward (CER), which predicts terminal reward through behavioral evidence in a trajectory prefix. CER synthesizes adaptive rubrics specific to the current task and stage through experiences summarized from related historical tasks. In test-time scaling on SWE-bench Verified, CER improves RM@8 over the strongest baseline by 4.2 percentage points (pp) on Nemotron 3 Ultra and 2.0 pp on Qwen 3.6 27B; on Nemotron, it takes only 15.3% tokens to match the best baseline performance. In RL training experiments, CER exceeds full-rollout TMax by 1.9 pp while using 52.7% fewer online policy-and-judge tokens. Together, CER provides an interpretable, efficient, and dense evaluation method for long-horizon coding agents.