Summary
Generating and scoring content from AI engines can be expensive and unreliable because current evaluation methods depend on costly and variable oracles. The authors propose a tested way to create a cheaper and consistent content score that doesn’t rely on these expensive sources. They show that older methods of validation no longer hold up with modern AI engines, and their approach detects when scores can be gamed or manipulated. Their protocol helps ensure the content score reflects quality rather than just popularity or citation signals. The solution also works alongside existing spam filters to improve content evaluation.
What this means in practice
- •For content platform operators: Implement a reliable, manipulation-resistant content score to better evaluate generative AI outputs without frequent API costs.
- •For spam detection teams: Integrate the score with existing spam filters to reduce vulnerabilities to out-of-distribution attacks in AI-generated content detection.
Abstract
How do you validate a cheap, deterministic proxy for an oracle that is expensive, rate-limited, and non-stationary? We present a protocol built on adversarial falsification gates (negative control, dose response, bounded amplification, duplication penalty, length neutrality) that define and select the proxy, fitted on a training split and confirmed held-out; around them it bounds what the proxy can never resolve, and re-measures external causal evidence on the current oracle rather than assuming it. We demonstrate it end to end on Generative Engine Optimization, where the proxy is a deterministic content score, and one step fails on that domain exactly as the protocol is built to detect: re-measuring the only published causal anchors (2023 effect sizes) on ten modern engine families shows their levers move citation on none, so the anchors are an expired external check; recalibrating to the near-zero modern vector strips the score of its lever-responsive components. What survives is the gate-enforced response surface. The gates buy a measured property: on a 500-source benchmark of adversarial edits, amplifying the score's calibrated levers gains an attacker at most 6 points, and decreases with dose; single-lever amplification is provably bounded, while the cap and cross-lever sub-additivity are empirical findings consistent with it. On detection, web-spam baselines dominate and out-of-distribution attacks evade the score, so the deployable filter layers it over them. A query-conditioned skyline bounds the score's citation signal (within-query Spearman 0.11), repositioning query-agnostic scores as quality filters rather than citation predictors. A query-leakage bug in our first ranking evaluation and a failed confidence flag are disclosed and corrected; every number reproduces offline from released artifacts at zero marginal API cost.