Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency
2026-08-17 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors studied how using one language model to check tasks and another (or the same) to fix them affects error reports. They found that having a completed audit and repair process already in the model's context reduces false alarms compared to when no prior audit was done. This reduction happens even when the previous audit found an error, which goes against some expectations. Further analysis shows that this change is due to the model adjusting its decision threshold, not its ability to distinguish errors. The authors also confirmed the findings remain consistent with reasoning enabled.
language modelsfalse alarmsauditrepairsignal detection theorydecision thresholdProcessBenchmodel contextreasoningerror checking
Authors
Parsa Mazaheri, Kasra Mazaheri
Abstract
Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer. We ask whether that wiring changes what the checker reports. Measuring false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical, we find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations, by 2.8 to 11.5 percentage points against a length-matched non-audit control, a 9 to 25% reduction relative to that control. The direction contradicts what the accumulated-message literature predicts: an episode whose audit reported an error lowers false alarms further still, at all five wordings on the model where that manipulation lands cleanly, though a negativity asymmetry predicts more flagging. Decomposing the episode finds repair content and audit verdict complementary: different components carry the effect on different model families. Signal-detection analysis locates the change in the threshold rather than in discrimination -- the criterion moves in 15 of 15 combinations and survives correction in 13 while d' survives in none, though the d' test is half as sensitive by construction -- and a hand audit of 50 false alarms finds 82% simply wrong, so at this operating point the shift need not be harmful. With reasoning enabled the effect keeps its relative size on both models tested, and the threshold reading holds there too.