Learning to Follow In-Context Watermark Instructions via Self-Distillation
Artificial Intelligence
Summary
The authors studied a way to hide signals (watermarks) in the answers of large language models (LLMs) by giving special instructions called in-context watermarking (ICW). They created a test called ICWBench to see how well different LLMs follow these instructions without ruining answer quality. Finding that current models struggled, the authors developed a two-step training method that improves watermark detection while keeping answers good. This method uses the same model teaching itself and then reinforcement learning with automatic checking. Applied to two models, their approach greatly boosted signal detection accuracy without hurting response quality.
Authors
Yepeng Liu, Tianyi Chen, Xuandong Zhao, Dawn Song, Yuheng Bu
Abstract
In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response. It thus equips LLMs with a watermarking interface that third parties can invoke without access to model internals. Its reliability hinges on the LLM following the instruction without degrading answer quality, yet how well current LLMs do so has not been measured. We introduce $\mathsf{ICWBench}$, a benchmark of three verifiable ICW instruction families, each scored on both detectability and answer quality. Evaluating 14 frontier proprietary and open-source LLMs, we find that none of the evaluated LLMs achieves both objectives across all three families. To address this, we propose a self-contained two-stage training method, requiring no distillation from a stronger model, no manual annotation, and no pre-existing ICW IF ability. The first stage, self-distillation with logits perturbation (SDLP), uses the same base LLM as both teacher and student: an instruction-equivalent decoding-time logits perturbation makes the teacher follow the ICW instruction, and the student is trained to match the teacher's output distribution. The second stage applies reinforcement learning with the automatic verifier as the reward. Applied to Qwen3-14B and GPT-OSS-20B, our method raises average TPR@$1\%$FPR across three ICW instructions from $0.100$ to $0.974$ and from $0.337$ to $0.968$, respectively, while maintaining high response quality under both perplexity evaluation and LLM-as-a-Judge.