Input assurance improves accuracy in evaluating AI agent actions
Does the Judge See What the Evaluation Requires? Input Assurance for LLM Agent Evaluation
Software Engineering
Summary
Evaluating AI agents that perform tasks involves checking their steps recorded as text packets. The paper finds that sometimes these packets don't include all needed details, causing the evaluator (judge) to misread or miss errors. The authors propose four rules to ensure the evaluator sees exactly what the evaluation requires, and test these rules on a sample system. Their approach helps catch mistakes earlier and makes judging more reliable.
What this means in practice
- •For ai system testers: Ensure that AI agent evaluation systems correctly capture and present all required input to judges, improving reliability of test results.
- •For software pipeline engineers: Integrate input-assurance tests into automated pipelines to detect and prevent missing or altered data during AI agent evaluation.
Authors
Boming Xia, Qinghua Lu, Runze Li, Dino Sejdinovic, Liming Zhu
Abstract
An LLM judge assesses agent steps from text packets. A renderer converts recorded runs into these packets, while software checkers test records and packets against requirements. We ask what evidence shows a packet is what the evaluation requires and propose four input-assurance obligations: required content survives rendering, checkers cover their requirements, a failed check stops later use and the judge reads the bytes last checked. In one pipeline with 30 trajectories, matching hashes, passing checkers and successful staging (copying packets for the judge) coexisted with violations of all four obligations, the last two in integration tests. The renderer omitted every structured tool call, leaving 53 of 60 judged steps blank despite matching hashes. For one judge configuration, later context raised error flags by 13 more on error than on correct steps. Restoring every call reduced this contrast to 2, a reduction of 11 (0.37 per trajectory, 95% cluster-bootstrap interval 0.17 to 0.53). In packets ending at the judged step, that configuration abstained at least once on every blank step, while the other four sometimes abstained but also flagged blank error and correct steps. Both separately written checkers accepted an unmatched tool result. In integration tests, a stub replacing the judge read a failed build's packet and a copy changed after staging. Three added controls stopped 18 of 19 constructed fault cases before the stub read any packet. The remaining stale script never ran. We turn the obligations into four tests and plan to evaluate their transfer to independent pipelines.