Agent verifiers can be misleading under strong search pressure
When Can You Trust an Agent's Own Verifier? Stress-Testing Counterfactual Rationale Verification Under Search Pressure and Oracle Access
Artificial Intelligence
Summary
Sometimes a computer program tries to improve itself by measuring how good it is with a special score. But if it tries too hard to improve that score, the score can stop being a true measure of quality. This paper studies how reliable an agent's own explanations are when it optimizes to satisfy its own verifier. The authors create tests to find when the verifier fails, and show examples where the verifier can be tricked or give misleading signals. They suggest checking verifiers carefully before trusting them.
What this means in practice
- •For machine learning engineers: Use stress tests to check if automated pipeline design tools rely on trustworthy internal verification scores before deploying them.
- •For software qa teams: Audit self-verifying AI agents with pressure ratios and failure probes to detect when their internal checks become unreliable during optimization.
Authors
Jonghyun Lee, Moonsu Kim, Sunghwan Park
Abstract
Automated agent design increasingly trusts a single score to decide whether a change is an improvement -- but a score optimized aggressively enough ceases to measure what it was meant to. When an agent also explains its design, that explanation is a second signal we can check without human labels: negate each stated claim, re-run, and observe whether performance drops. We ask whether such a verifier stays reliable once an agent optimizes against it. Our contribution is not a new verifier but a way to stress-test one: a pressure ratio $ρ$ measuring how much the verifier steers the agent's search, and four falsifiable probes covering four failure modes -- giving no signal, being too weak to matter, being gamed by stating fewer claims, and being queried before the agent commits. We apply the test to our own verifier, the Rationale Consistency Score (RCS), inside an LLM agent that designs machine-learning pipelines, with one generator model, six tasks, and three seeds. First, in our prior configuration the verifier barely affects the search objective ($ρ\ll 1$), partly because of an accuracy gate, so its robustness had not been tested; it still shapes the reported artifact through a final-selection tie-break that $ρ$ does not register. Second, a capability control shows the agent shortens its claim lists when instructed, yet search does not, even at $ρ$ over 100 times the prior level; since feedback omits claim volume and horizons are short, this null result is largely structural rather than evidence of robustness. Third, when the agent can probe the verifier before committing, it keeps claims that pass while held-out faithfulness falls -- a decline invisible to in-loop metrics, though partly confounded by how RCS scores small pipelines. We offer these as preliminary evidence and propose a reporting standard: audit a verifier with $ρ$ and these probes before trusting it.