Detailed traces shift how AI overseers judge language model outputs
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers
Computation and LanguageArtificial IntelligenceComputers and SocietyHuman-Computer Interaction
Summary
Organizations often use one AI language model to check the work of another, looking at the steps claimed along the way. The authors found that giving too much detail in these step-by-step traces can make the checking AI more likely to wrongly reject good answers. This happens because detailed traces change how strictly the AI judges the work, not because it misses errors. Labels that explain choices help reduce some mistakes, but even then, overseers sometimes incorrectly reject correct answers when detail grows. So, the authors suggest evaluating AI overseers not just by accuracy but also by how their decision style changes with extra information.
What this means in practice
- •For ai safety teams: Adjust AI auditing systems to monitor false rejection behavior influenced by procedural trace detail for more reliable oversight.
- •For automated compliance units: Improve compliance checking workflows by tuning trace detail and labeling to reduce false alarms when using AI overseers.
Authors
Zihan Chen, Di Zhu, Lei Zheng, Weiling Li
Abstract
Organizations increasingly use oversight loops where one large language model (LLM) audits another's outputs alongside procedural traces of claimed steps. A common concern about such LLM-as-a-judge pipelines is that detailed traces make overseers gullible. Using signal detection theory, we audit five LLM overseers on 19 compliance tasks (4,551 analyzed judgments), varying only trace detail and evidence labeling. With disconfirming evidence always visible, error detection remains near ceiling. Instead, elaborate traces shift the decision criterion toward rejection, increasing false alarms in susceptible overseers. Without option labels, human-validated reason coding shows about 60% of false alarms cite an inability to tie evidence to its option. Labels eliminate this stated reason, yet residual rejection of correct work persists in those overseers and rises with trace detail. Procedural traces thus act as governance artifacts that shape oversight decisions. AI auditors should be evaluated by their decision criterion and false-alarm behavior, alongside accuracy.