Adversarial watermarks can disrupt document AI without ruining readability
Visually Legible but Adversarial: Watermark-Constrained Attacks on End-to-End VLMs for Document Understanding
Computer Vision and Pattern Recognition
Summary
Documents often have visible watermarks to show ownership or confidentiality, but these can also be used to confuse AI systems that read documents. The authors created a method called Auto-Regional Adversarial Watermarking that changes watermark pixels in a way that tricks visual language models, making them misread or misunderstand the document. This trick works without needing to see or use any questions or answers and still keeps the watermark easy to read for people. Their tests showed this approach greatly reduced the accuracy of document question-answering AI while keeping the watermark legible.
What this means in practice
- •For document ai engineers: Evaluate and harden AI models against watermark-based adversarial attacks to improve reliability in document processing pipelines.
- •For security assessment teams: Use watermark-shaped adversarial attacks to test vulnerabilities of document understanding AI before deployment in sensitive environments.
Authors
Qingzhe Han, Fan Dang, Shengqi Chen, Xinyi Sun, Haoyu Xiong, Lingkun Li, Di Zhang
Abstract
Visible watermarks convey document ownership, confidentiality, and status, but can also carry adversarial signals. We introduce Auto-Regional Adversarial Watermarking (AR-AWM), a model-aware white-box attack on end-to-end visual language models (VLMs) for document understanding. AR-AWM constructs a small, spatially distributed watermark-text support and optimizes only its pixels to disrupt visual representations. The attack is untargeted and question-agnostic: generation uses no questions, answers, parsing prompts, language-model gradients, or language-model outputs. Each attacked page is reused across downstream queries. On Qwen2.5-VL-7B, a watermark support covering 7.98% of page pixels under the reporting threshold reduces DocVQA exact match from 0.899 to 0.150 and average normalized Levenshtein similarity (ANLS) from 0.939 to 0.212. Selected qualitative cases retain visually legible source fields. The same support without optimization and with matched random perturbations retains ANLS of 0.932 and 0.933, respectively, identifying adversarial optimization as the primary cause of degradation. Layout-matched optimized rectangles yield ANLS of 0.276, showing an incremental benefit from text-shaped geometry. AR-AWM also degrades all reported text, formula, reading-order, and table metrics on OmniDocBench. Separate white-box optimization on InternVL2.5-8B reduces ANLS from 0.922 to 0.046, while weak cross-model transfer reveals strong encoder specificity. These results identify conventional watermark-shaped supports as an adversarial surface for document VLMs in the evaluated white-box setting.luated white-box setting.