Protocol tests if AI agents truly improve from experience traces
Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces
Artificial Intelligence
Summary
Checking if AI agents actually get better by learning from past experiences is tricky because sometimes systems look improved without changing. The authors introduce a way to measure real improvement by separating simple rescoring from actual updates to the AI's behavior. They test various update methods on sets of coding and office tasks, finding mixed results where some improvements appear but are not consistent across all tests. This helps clarify which methods genuinely evolve AI performance from their own past interactions.
What this means in practice
- •For ai system developers: Test AI agents for genuine improvement by separating rescore effects from true updates to strategies using the Trace Evolution Bench protocol.
- •For software testing teams: Use automated coding tasks with hidden tests to validate if software agents improve meaningfully after updates rather than appear improved by chance.
Authors
Fangfei Li, Chenyang Zhao, Long Wang, Feng Tian, Lv Guo
Abstract
Self-evolving agents are usually scored by endpoint Gain: held-out success after an update minus success before. That difference can credit a frozen rescore as evolution. We introduce Trace Evolution Bench (TEB), a protocol that separates that rescore from a context or weight update, together with three frozen 80/20/40 packs. An update is accepted only when validation does not fall, held-out identifiers never enter the update, and a text update is not ranked with a weight update. The primary scoreboard is coding, graded by hidden unit tests in a closed workspace. On that pack (40 held-out tasks, one family), DeepSeek v4 Pro under no_evolve never changes its policy yet moves 7.5, 5.0, and 12.5 percentage points across three seeds. Of nine text-update cells inspired by GEPA, ACE, and ExpeL, seven have positive excess above that seed's own band, and one gepa_llm seed moves down by the width of its band (-12.5, regression 0.48). GLM and MiniMax do not reproduce that seven-of-nine pattern. No method stays above its own band on every seed of all three models: GLM has two paired cells above their own bands, and MiniMax expel_llm seed 44 is a third (regression 0.30). The methods were fixed on a validation screen before held-out was read. Office, graded by rule-bounded delivery, and manufacturing table QA are separate packs: office DeepSeek does not exceed its own bands (widest 17.5 points), and manufacturing has no paired positive excess beyond its seed band.