Text watermarking causes large language models to make factual errors
Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate
Cryptography and SecurityArtificial Intelligence
Summary
Watermarking is a way to mark AI-generated text to tell it apart from human writing, but this paper finds that it can sometimes make AI models give wrong facts even when they have the right information. The authors show that several common watermarking methods cause these mistakes by changing word choices slightly or distracting the model as it writes. They also suggest two fixes that reduce these errors a lot without losing the ability to detect watermarked text. This work points out that checking for truthfulness should be as important as spotting watermarks when using these methods.
What this means in practice
- •For ai platform engineers: Improve watermarking in text generation systems by applying new methods to reduce induced factual errors while maintaining watermark detection.
- •For content moderation teams: Use insights from watermark-induced errors to better balance detection of AI content and verification of facts in user submissions.
Authors
Haocheng Ye, Aoting Hu, Xinwei Zhang, Xunzhu Tang, Shuchao Pang, Jason Xue
Abstract
Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled retrieval-augmented generation setting, we compare unwatermarked and watermarked generations under the same context, query, and decoding configuration, and quantify their factual accuracy decrease. Across six representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID watermarking, we consistently observe watermark-induced hallucination. Watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) token perturbations in the current-step arising from method-specific reweighting or keyed sampling, and (2) prefix-induced attention drift, which accumulates through autoregressive decoding and weakens later attention to the factual context. Motivated by this analysis, we propose two plug-in interventions at the token and attention levels that can be integrated into existing watermarking methods to improve factuality. At a matched TPR of 0.90 at 1% FPR, combining the two interventions reduces factual errors by approximately 90% relative to watermark-only decoding while preserving fluency and comparable decoding efficiency. Overall, this work highlights factuality as a first-class criterion in watermark evaluation, alongside detectability and robustness, and calls for careful factuality validation before deploying watermarks in fact-critical applications.