AgentReplay improves fairness in AI agent performance testing
AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking
Distributed, Parallel, and Cluster Computing
Summary
Evaluating how well AI agents perform is tricky because small differences in their generated responses can change the whole process they follow. The authors created AgentReplay, a system that records every token (word or piece of text) generated by an AI agent and then replays the exact same sequence when testing different setups. This ensures that differences in performance come from the system itself, not from changes in what the AI says or does. AgentReplay also separates the task of creating test data from measuring performance, letting smaller AI models replay tasks made by bigger ones for fair comparisons.
What this means in practice
- •For ai infrastructure engineers: Compare and evaluate serving strategies fairly by replaying exact AI agent behaviors to isolate system improvements from workload differences.
- •For cloud service operators: Test and tune AI service configurations using consistent workloads recorded from high-capability models replayed on smaller systems for operational benchmarking.
Authors
Zaifeng Pan, Michael Wang, Chris Wu, Zhengding Hu, Xinwei Qiang, Zhongkai Yu, Yufei Ding
Abstract
LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimizations is challenging because identical tasks can produce different execution trajectories. Changes in generated tokens can alter subsequent prompts, tool calls, and reasoning turns, making it difficult to distinguish system improvements from workload variation. Greedy decoding does not guarantee identical outputs, while replaying only sequence lengths loses token information that affects prefix caching and mixture-of-experts (MoE) routing. To address these problems, we propose AgentReplay, a configurable trace record-and-replay framework for agent serving. AgentReplay records input/output tokens, expert selections, request dependencies, and tool durations. During replay, it forces the recorded output tokens while performing normal autoregressive computation, with optional controls for MoE expert selection and tool delays. This allows different systems to execute the same recorded workload while retaining their own batching, scheduling, and parallelization decisions. We further separate trajectory generation from performance evaluation, enabling compatible smaller models to replay long-horizon traces collected with more capable models. Our experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.