Verifier audits can favor worse policies when policy improves
Better Policies, Worse Verifier Audits
Machine Learning
Summary
When checks called verifiers are used to evaluate policies or decisions, improving one part of the policy can hide errors in another part. The authors show this can cause the verifier to give misleadingly good scores, even preferring worse verifiers. They demonstrate this problem happens in real settings, like medical signal analysis, and provide a fix: verifiers must audit failures directly rather than relying on mixed data. This ensures more reliable evaluations and prevents errors being hidden.
What this means in practice
- •For machine learning engineers: Improve reliability of policy verification loops by auditing component failures directly to prevent misleading success metrics.
- •For medical data analysts: Avoid performance degradation in predictive models by keeping failure cases in verification audits when updating model components.
Authors
Daniel Cieślak, Jacek Łubiński
Abstract
Verifier stacks increasingly sit inside post-training and test-time compute loops. We study a specific control loop: a component verifier is evaluated on fresh policy outputs against the full success label and refreshed when that aggregate score falls. For a conjunctive target $y = F \wedge M$, its on-policy AUC is $(1-q)A_{\mathrm{own}} + qA_{\mathrm{cross}}$, where $q = \Pr[F=1 \mid y=0]$. As the policy masters $F$, $q$ can rise toward one, removing weight from $F$-failures and rewarding information about $M$. We call this on-policy audit collapse. We derive a sharp identified set, a complete-erasure threshold, and the exact point at which the audit can prefer a worse component verifier; its population optimum is the full-outcome predictor, not the component oracle. An exact oracle exhibits the collapse, a natural multi-label text control shows inversion can select a head with lower canonical $F$-AUC at the observed failure mixture, and a prediction-registered ECG study shows that acting on the audit can reduce off-policy $F$-AUC, transfer the damage to an unseen generator, and increase downstream constraint violations. Preserving own-failure support prevents the damage. Matching label-complexity bounds and a controlled-mixture audit give the corresponding repair: audit the component on its own failures, and keep those failures in the refresh distribution.