BERT can generate text iteratively without special training
BERT Is Already a Denoiser: Cheaply Unlocking Iterative Generation in Masked Language Models
Computation and Language
Summary
Masked language models like BERT were mainly used to understand text, but the authors found that BERT can also fix missing words repeatedly to generate new text sequences. They showed that BERT’s original training already enables this denoising ability without needing special diffusion training. By adapting BERT further for specific tasks, it can generate more accurate and meaningful text. This means that the different stages—pretraining, adapting, and decoding—work together for effective text generation without extra complex training.
What this means in practice
- •For natural language processing engineers: Use pretrained masked language models for iterative text generation without adding diffusion training, making generation simpler and more efficient.
- •For chatbot developers: Improve chatbot responses by adapting existing masked language models for task-specific iterative generation using fewer resources and simpler methods.
Authors
Julio C. Amador Diaz Lopez
Abstract
Masked language models (MLMs) such as BERT are typically used for representation learning despite their denoising pretraining. We test whether this capability supports conditional generation without diffusion training. Our decoder repeatedly predicts masked tokens and commits high-confidence predictions using a cosine schedule. It requires no diffusion timestep or reverse-diffusion objective. Frozen BERT already exhibits substantial denoising capability, reconstructing 61.0%, 37.5%, and 24.4% of masked tokens at 15%, 45%, and 60% corruption, respectively, although generation deteriorates sharply as visible context disappears. This capability alone does not yield task-directed generation: frozen BERT reaches only 11.0% on ARC-Easy and 4.2% on MMLU. Ordinary MLM adaptation, however, rapidly converts the same architecture into a usable generator. Full adaptation reaches 28.2% and 24.6%, respectively, while parameter-efficient adaptation substantially reduces unmatched outputs. On open-ended conditional generation, full MLM adaptation reaches BLEU/ROUGE-L of 0.163/0.583 on QQP and 0.107/0.327 on Quasar-T, placing it in the same broad performance range as published BERT models trained through diffusion without explicit diffusion training. Finally, applying the same generative interface to stronger masked representations substantially improves performance: RoBERTa-large reaches 44.2% on ARC-Easy and 33.8% on MMLU. These results separate three components of masked iterative generation: pretraining supplies the representation and denoising operator, task adaptation determines what the model generates, and an iterative commitment rule supplies the decoding process. Explicit diffusion training is therefore not necessary to obtain effective iterative conditional generation from pretrained MLMs in the settings studied here.