Summary
Measuring how well AI content generators perform is hard because the best tests are slow, expensive, and change over time. The authors created a step-by-step method to check a cheaper, predictable scoring system that estimates quality without running the full, costly tests. Their method uses special checks to catch when the score might be manipulated or unreliable and adjusts the score based on what still holds true today. They found that older benchmarks no longer work for modern AI engines and showed that their score resists being tricked while still giving useful quality feedback. They also fixed some early mistakes in their testing and provided all materials so others can reproduce their results cheaply.
generative enginescontent scoreoracle validationadversarial falsificationproxy metricscausal anchorscalibrationquery conditioningreproducibilitymanipulation resistance
Abstract
How do you validate a cheap, deterministic proxy for an oracle that is expensive, rate-limited, and non-stationary? We present a protocol built on adversarial falsification gates (negative control, dose response, bounded amplification, duplication penalty, length neutrality) that define and select the proxy, fitted on a training split and confirmed held-out; around them it bounds what the proxy can never resolve, and re-measures external causal evidence on the current oracle rather than assuming it. We demonstrate it end to end on Generative Engine Optimization, where the proxy is a deterministic content score, and one step fails on that domain exactly as the protocol is built to detect: re-measuring the only published causal anchors (2023 effect sizes) on ten modern engine families shows their levers move citation on none, so the anchors are an expired external check; recalibrating to the near-zero modern vector strips the score of its lever-responsive components. What survives is the gate-enforced response surface. The gates buy a measured property: on a 500-source benchmark of adversarial edits, amplifying the score's calibrated levers gains an attacker at most 6 points, and decreases with dose; single-lever amplification is provably bounded, while the cap and cross-lever sub-additivity are empirical findings consistent with it. On detection, web-spam baselines dominate and out-of-distribution attacks evade the score, so the deployable filter layers it over them. A query-conditioned skyline bounds the score's citation signal (within-query Spearman 0.11), repositioning query-agnostic scores as quality filters rather than citation predictors. A query-leakage bug in our first ranking evaluation and a failed confidence flag are disclosed and corrected; every number reproduces offline from released artifacts at zero marginal API cost.