Semantic watermarking detects manipulated images and how they changed
Semantic Watermarking for Malicious Image Manipulation Detection
Computer Vision and Pattern Recognition
Summary
Images can be secretly altered by AI to add harmful content while looking real, which can mislead people. The authors created a method that hides special signals called semantic watermarks inside images, making it possible to tell if and how an image was changed. Their approach uses a smart encoding system that can still recover these signals even if the watermark gets damaged. They also built a tool that reads these signals to explain the type of change made to the image, which is not possible with simpler watermark methods.
What this means in practice
- •For digital content moderators: Detect altered images and identify the type of semantic changes to improve content review accuracy in online platforms.
- •For forensic analysts: Use recovered semantic watermarks to trace direction and nature of image manipulations for legal or investigative purposes.
Authors
Yoonseo Kim, Seungwoo Baek, Junyoung Park
Abstract
The proliferation of high-fidelity generative editing models has made it possible to inject violent or sexual content into otherwise ordinary images while preserving visual plausibility, with concrete consequences for public discourse and vulnerable populations. We propose a robust semantic watermarking framework that reframes the watermark as a recoverable semantic reference rather than an opaque identifier. Our framework combines a $β$-VAE-based binary watermark (CLIP-VAE) with explicit channel-aware training---random bit-flip noise is injected during training so that the decoder learns graceful degradation under the noisy watermarking channel. As a downstream application, a lightweight module SDA-Net uses the recovered semantic embedding to expose not only whether but in which semantic direction an image has been altered. In a 5-way comparison against representative binary hashing baselines (SimHash, ITQ, HashNet, and their robust-MLP variants), CLIP-VAE achieves the highest reconstruction cosine similarity to the original CLIP embedding under realistic InstructPix2Pix bit-error rates, and uniquely supports direction-of-drift detection---a forensic complement to existing content-moderation pipelines.