Reproducible evaluation improves prompt compression for chat agents
Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents
Artificial Intelligence
Summary
Evaluating AI chat agents that work with constantly changing business data is tricky because the data updates can make past tests irrelevant. The authors created a testing setup that can be repeated reliably even as live data changes, and can be refreshed when outdated. They then used this setup to safely shrink the AI prompts without losing accuracy by preventing tricks that might cheat the test. When tried on a real merchant service, this method reduced prompt size by up to 7.6% without hurting task performance, sometimes even improving correctness.
What this means in practice
- •For enterprise ai teams: Maintain stable benchmarks for conversational agents interacting with live changing business data by using reproducible offline evaluation.
- •For natural language processing engineers: Shrink AI prompt sizes safely without losing accuracy by using correctness-gated prompt compression driven by reproducible evaluations.
- •For customer support platform developers: Develop more efficient, accurate chatbots for merchant-facing applications by applying prompt compression methods validated with stable evaluation setups.$Commercial implications: Enables building better performing and smaller prompt chatbots for commercial customer service solutions, improving scalability and costs.
Authors
Prabal Vashisht, Zheng Hu, Ying Yang
Abstract
Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consistently across three independent subject-model families, and for one, correctness measurably improves.