New method improves watermark detection in language model outputs
Predictive Likelihood Ratios for Language Model Watermark Detection
Artificial IntelligenceMachine Learning
Summary
Detecting hidden watermarks in text generated by AI language models can be tricky, especially when details about the watermark are uncertain. The authors developed a new approach that combines multiple assumptions about these uncertainties to better detect watermarks without needing exact tuning. Their method controls false alarms even when tests are stopped early and works well across different scenarios. Although it’s not perfect in all cases, tests show it generally improves accuracy over previous methods.
What this means in practice
- •For ai system developers: Implement watermark detection in AI-generated text systems to reliably identify flagged tokens without needing precise signal tuning.
- •For content moderation platforms: Incorporate robust text watermark detection to help verify AI-generated content and prevent misuse on user-generated platforms.
Authors
Li Ma
Abstract
Keyed watermark detection tests dependence between observed tokens and pseudorandom variables reconstructed from a secret key. Building on the pivotal framework of Li et al. (2025), we construct predictive likelihood ratios that average over uncertain probability deficits and residual-tail distributions. The aim is robust detection power across alternative specifications without requiring a single signal-strength tuning. A mixture prior combines tail shape and effective width; hierarchical extensions allow within-document variation in deficit or width. The test maximizes prior-averaged power at a fixed size, but is not generally uniformly most powerful or minimax. Under the exact conditional pivot null, normalized predictive alternatives selected before each observation yield a Bayes factor that is also a test martingale: Type I error control is unaffected by alternative misspecification and remains valid under optional stopping. This guarantee does not cover violations of the conditional null, and the interpolated implementation has no certified anytime guarantee. Gumbel marginal likelihoods are evaluated by fixed quadrature. Across the evaluated tail-shape and tail-width alternatives and three horizons, the union-tail mixture has maximum observed Type II error regret .0080, compared with .0962 for the equal-tail mixture, relative to the best tested rule. On temperature-matched outputs from two open models, it improves AUC over the equal-tail baseline in all eight non-saturated model-temperature cells, although the leading reference score generally has higher AUC. Supplementary experiments show retained power under independent null-like replacement and smaller changes from hierarchical dependence modeling. The evidence supports robustness across the evaluated alternatives, not uniform power guarantees or resistance to arbitrary text edits.