Language agent training improves credit assignment with selective replay and evolving rubrics
CREDO: Variance-Guided Rubric Evolution for Replay-Corrected Credit Assignment
Artificial Intelligence
Summary
Training language agents to perform complex tasks is hard because feedback often only comes at the end, making it difficult to know which actions helped. The authors propose Credo, a method that uses intermediate progress assessments combined with carefully chosen replay of past actions to better assign credit to useful steps. Credo updates its scoring rules over time and decides when and what to review to improve training efficiency. This helps the agent learn more reliably without needing to replay everything exhaustively.
What this means in practice
- •For ai training engineers: Incorporate Credo’s selective replay and evolving rubrics to improve training efficiency in language agent development environments with sparse feedback.
- •For game ai developers: Use Credo’s credit assignment method to better evaluate intermediate agent actions without costly exhaustive replays in resettable training simulations.
Authors
Xuchun Hu
Abstract
Long-horizon language agents receive sparse terminal feedback, while intermediate rubrics provide structured but potentially misspecified assessments of progress. In resettable training environments, counterfactual continuation rollouts can measure local credit, but exhaustive replay is costly. We propose Credo, a framework that couples evolving semantic rubrics with selective, execution-based credit correction. A frozen judge maps visible transitions to rubric features, and a credit head predicts the change in expected terminal reward associated with the realized transition. Independently sampled two-sided replays correct prediction residuals using their recorded inclusion probabilities. We derive conditional unbiasedness and a variance decomposition that connects two design choices: which rubric features to retain, and where to allocate a fixed expected replay budget. The resulting criterion weights prediction errors by policy-score sensitivity and missing replay coverage; its allocation rule additionally accounts for continuation cost. We also describe a practical mixture with terminal leave-one-out advantages and distinguish its clipped, token-normalized PPO implementation from the ideal policy-gradient estimator. This preliminary report provides the method, proofs, an exact finite-model audit, and a controlled evaluation protocol. It makes no claim of empirical superiority on language-agent benchmarks.