Copygen improves exact span reproduction in large language models
When Copying Is Hard: Copy-Constrained Decoding for Exact Span Reproduction
Computation and Language
Summary
Sometimes language models struggle to copy exact pieces of text from a large amount of information, even when the text appears there. This is a problem for tasks like writing code with exact keys. The authors created a test to measure this problem and found current models make many copying mistakes. They designed CopyGen, a new technique that helps models decide when and how much to copy exactly, making copying much more reliable without changing the model itself. CopyGen improves exact copying accuracy significantly and speeds up text generation.
What this means in practice
- •For software developers: Embed exact sensitive strings like API keys into generated code snippets with much greater reliability and speed.
- •For legal document processors: Ensure verbatim copying of contract clauses to maintain legal accuracy in automated document handling.
Authors
Jinghui Zhang, Lang Gao, Zongfang Liu, Ruihong Zeng, Zirui Song, Rui Yan, Kentaro Inui, Xiuying Chen
Abstract
Precisely reproducing a target span verbatim within a long context is a fundamental capability test for LLMs: even when the target string is present in context, models often still drift within the span or fail to stop correctly, revealing limitations in source grounding and boundary control. This capability has broad practical relevance. For instance, code generation often requires embedding sensitive strings such as API keys exactly as they appear in context. In this work, we construct a benchmark to test exact span reproduction ability across contracts, scientific literature, code, and random sequences, and find that current models still make surprisingly frequent errors. To address this, we propose CopyGen, a lightweight copy-constrained plugin for frozen LLMs that turns copying from a byproduct of next-token prediction into a controlled process. Our key idea is to decompose copying into three distinct decisions: when to copy, how far to copy, and when to stop. Concretely, CopyGen first determines whether to enter copy mode, then predicts a tentative copy length, and finally verifies the generated span token by token. During verification, we adaptively adjust the logits of copy continuation tokens to incentivize or suppress continuation, and emit only the longest valid prefix. Across three LLM backbones and four benchmark tasks, CopyGen improves average exact match by 35% relative without model finetuning, and integrates seamlessly with SFT to further improve it by 4%, while substantially accelerating generation. These results suggest that reliable exact copying is not something LLMs obtain for free, but a decoding-time control problem that requires explicit modeling. Code is available at https://github.com/mbzuai-nlp/CopyGen.