Chronicle creates repeatable tests for large language model agent failures
Chronicle: Cut-Point Replay for Regression Testing of LLM Agents
Computation and LanguageArtificial Intelligence
Summary
Large language models can give different answers each time they run, making it hard to recreate and fix mistakes made by programs using them. The authors present Chronicle, a tool that records key points where the model’s randomness affects the run and can replay these while running new code on other parts. This makes it possible to turn a past failure into a reliable test that checks if code changes fix the problem. They show Chronicle works well on simulated data and can catch more errors than simpler testing methods.
What this means in practice
- •For software engineering teams: Run reliable regression tests on AI agent failures to catch bugs caused by changes to code that interacts with language models.
- •For ai platform developers: Integrate cut-point replay testing into continuous integration pipelines for improved quality control of AI agent updates.
Tested on simulated data.
Authors
Tisha Chawla, Susheem Koul
Abstract
Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitwise reproducible, on tools that read changing state, and on a multi-step trajectory that a re-run rarely repeats. Record-and-replay makes a run reproducible, but existing agent tooling records runs only to trace or score them, not to test a code change against them. We present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays it from the record. Its central operation, cut-point replay, serves a chosen subset of boundaries from the record and executes the complementary subset live with new code, turning a recorded incident into a regression test that runs in continuous integration. On a benchmark of 6 recorded failures with simulated model boundaries, recording adds 23 μs per crossing (0.008% of an assumed 300 ms model call), full replay issues zero model calls and is bit-stable across 20 repetitions, and cut-point tests fail on faulty code and pass on guarded and benign changes for all 6 incidents. In a mutation study of the guarded tools, cut-point tests catch every mutant that lets the recorded unsafe action through, while a baseline that stubs every boundary, using the same assertion, catches none. Chronicle and the benchmark are publicly available at https://github.com/theagentplane/chronicle.