Papers for

ai ops teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Cargo improves ai evaluation by using context to avoid false errors

CARGO: Context-Aware Retrieval-Gated Evaluation of Agentic AI in Production

Abstract: Reference-based LLM-as-a-judge evaluation assumes the reference answer is the target. In deployed agentic systems that operate over dynamic entities (support cases, assets, accounts), the closest available reference typically applies the correct procedure to a different entity, so a literal judge penalizes different identifiers, dates, and statuses as errors or hallucinations. We name this failure mode reference-instance divergence (RID). We propose CARGO, a framework that (i) treats retrieved references as procedural exemplars and grounds factual judgments in the live instance's observed context, (ii) assigns each claim a three-way status (supported, contradicted, unverifiable) and penalizes only contradictions, and (iii) gates evaluation by retrieval confidence, casting production evaluation as selective prediction. We introduce CARGO-Bench, a perturbation-based diagnostic suite with ground truth by construction that separates leniency from discrimination. On CARGO-Bench (246 items, two judge models, 7,872 judgments), the standard reference-based judge penalizes 100% of correct entity-transplanted answers and is uninformative (discrimination index DI ~ 0); supplying the live facts without reframing changes nothing. CARGO eliminates these false penalties (0/50) while retaining near-complete contradiction recall (50/50 and 49/50), raising DI to 0.58 [0.48, 0.68]; a rubric-swap control attributes most of the effect to context-grounded dimension definitions. CARGO also exposes a limitation of its own design: the leniency that protects entity values suppresses detection of procedural corruptions (20% recall). A post-hoc fix does not close the gap, and an LLM-as-annotator study with written guidelines and adjudication shows the same blind spot. We release a preregistered protocol for extending the evaluation to expert agreement, risk-coverage, and cost on production traffic.

Thu 24 SeptComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Evaluating AI systems that handle live data like support cases can be tricky because the reference answers usually refer to different cases with different details. The authors show that traditional evaluation methods wrongly mark correct answers as mistakes due to these differences. They propose CARGO, a way to judge AI outputs by considering the actual context and only penalizing true contradictions. Their tests show CARGO avoids false penalties while still catching real errors, although it struggles with some types of mistakes related to procedure.
Open → 2609.30471v1