Lightweight method improves diagnosis of AI agent failures with less cost
Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents
Artificial Intelligence
Summary
AI agents often make mistakes during their tasks, and figuring out why requires checking detailed records of their actions, which can be slow and expensive. The authors designed a new lightweight way to quickly review these task traces using smart rules and a single AI judge, focusing only on the important parts. Their approach finds problems more accurately and much faster than older methods, making it useful for real-world AI systems. It has already been used successfully in an enterprise platform.
What this means in practice
- •For enterprise ai teams: Lower the cost and speed up the diagnosis of agent failures in production AI systems using rule-guided evaluation summaries.
- •For customer support software developers: Improve monitoring by quickly identifying where automated helpers fail in their dialogue or tool use.
Authors
Linh-An Phan, MingXue Wang, Guangyu Wu, Feng Pan, Zhaoyu Pang, Yanbin Zhang
Abstract
Trajectory evaluation is essential for improving the reliability of LLM-based agents, but production use makes it expensive to run repeatedly. Modern agents generate long traces containing tool calls, observations, retries, and external outputs, while not all raw tokens are equally useful for diagnosis. We present \textit{LiteTrajEval}, a lightweight architecture for budget-bounded trajectory evaluation. LiteTrajEval derives compact domain-specific rule profiles offline, then preprocesses each trajectory online, marks heuristic failure signals, serializes it under a fixed global budget, and invokes a single rubric-guided LLM judge to produce structured diagnostic reports. Evaluated on public Magentic-One-style and $τ$-bench-style trajectory datasets, LiteTrajEval improves failure-localization alignment with human annotations by roughly 20--35 percentage points on Magentic-One and up to 23 percentage points on $τ$-retail compared with AgentRx, while reducing cost by about 6$\times$ and evaluation time by more than 8$\times$. This solution has also been deployed in our enterprise agentic platform.