Decoy-Calibrated Failure Audits for Language Models
2026-06-08 • Machine Learning
Machine LearningComputation and LanguageInformation Retrieval
AI summaryⓘ
The authors present Janus, a method to check if explanations for a model's errors are truly reliable or just random coincidences. They start with a set of possible error causes (descriptors) and compare real error patterns to fake ones created at random. An explanation is only accepted if it shows a strong error pattern both initially and on new data. Testing Janus on tasks and benchmarks, the authors show it helps avoid false alarms by confirming only credible error explanations. This approach helps auditors separate guessing potential issues from confidently reporting real problems.
Model auditingError analysisError-rate liftDescriptorsHoldout dataStatistical significanceMulti-table lookupFalse discovery controlLarge language models (LLMs)Benchmark datasets
Authors
Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh
Abstract
Useful audits reveal not only how often a model fails, but also where its failures concentrate. An auditor may test many candidate explanations: long inputs, indirect questions, distracting evidence, or combinations of these factors. The risk is selection. The largest observed effect may reflect a real failure mode, or it may simply be the best result among many tried. We introduce Janus, a procedure for deciding when a proposed error explanation is credible enough to report. The goal is not to generate new explanations, but to decide which ones hold up. The auditor starts with a fixed model, a labeled evaluation set, and a frozen list of candidate explanations, which we call descriptors. Janus scores each descriptor by its error-rate lift, then compares real descriptors with fake ones that have the same frequencies but are randomly assigned to examples. A descriptor is confirmed only if it beats this decoy floor on the data used for discovery and then repeats on separate held-out data. In a controlled audit of multi-table lookup tasks, Janus identifies the planted failure, confirming long-chain descriptors and their interactions. The LLM often stops partway through the lookup chain instead of reaching the final answer. On two public benchmarks, MuSiQue and LongBench v2, the SliceLine baseline flags plausible high-error pockets, but Janus confirms none of them. Ablations show why both safeguards matter. On LongBench v2, an uncalibrated fixed threshold reports 20 descriptors, the decoy floor leaves one, and the holdout check rejects the last one after its lift shrinks from 0.36 to 0.05. The resulting principle separates proposing explanations from reporting them. Candidates may come from any source, but only those that beat decoys and replicate on fresh data become audit findings.