AuditKIT unifies large language model and agent evaluation metrics
AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems
Computation and LanguageArtificial IntelligenceInformation RetrievalSoftware Engineering
Summary
Evaluating language models and AI agents is complicated because different tests measure different things in different ways, making it hard to compare results or find mistakes. The authors created AuditKIT, a tool that collects all evaluation details into one system that can handle different types of tests together. This tool can watch tools the AI uses, check if answers are correct, and handle uncertain or missing information carefully. They showed through examples that how metrics and runtimes are chosen can change evaluation results a lot.
What this means in practice
- •For ai development teams: Assess language models and agent behaviors uniformly to compare and improve AI systems across benchmarks and tool use.
- •For software testers: Track and evaluate runtime performance and task success in complex AI workflows involving multiple tools or data sources.
Authors
Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth
Abstract
Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows. We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics. Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.