Efficient evaluation methods improve testing of evolving AI agents
Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
Artificial IntelligenceSoftware Engineering
Summary
Testing AI agents frequently is important but can be very expensive and time-consuming. The authors studied different ways to make repeated testing of a large AI agent more efficient using past test results. They found that a smart adaptive testing method worked best but chose a simpler fixed subset of questions for day-to-day use because it was easier to manage. This fixed method also worked well for other similar AI agents without needing changes. The paper shares advice from real-world experience on how to test AI agents in production regularly.
What this means in practice
- •For machine learning engineers: Run efficient recurring evaluations of AI agents to save resources while keeping accuracy high in production environments.
- •For software reliability teams: Maintain stable benchmarks for evolving software agents using difficulty-stratified fixed test subsets that require minimal recalibration.
Authors
Yining She, Lei Lin
Abstract
Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation for a production analytics agent serving tens of thousands of monthly active users and report first-hand deployment experience. Using 574 historical runs of the production benchmark, split chronologically into calibration and held-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The results show that multidimensional 2PL adaptive testing achieves the best overall score fidelity: executing 200 questions, 38.5% of a full run, yields 1.03 pp of MAE. We nevertheless deployed difficulty-stratified fixed subsets because of their operational simplicity, and show they transfer without recalibration to five other agent families and remain stable across calibration windows as short as one day. Drawing on this deployment experience, we report practical recommendations for recurring production-agent evaluation.