Masked diffusion models reveal detailed text evaluation signals
Rate-Resolved Text Evaluation with Masked Diffusion Language Models
Computation and Language
Summary
Evaluating how well computers write text usually gives a single score that ranks their output but doesn't explain why some text is better or worse. This paper shows a way to get more detailed feedback by masking different amounts of text and seeing how hard it is for a model to fill in the blanks. The authors find this approach provides new clues about where text generation systems struggle, helping to diagnose specific problems. This method depends on the specific model and task, so it needs testing for each use.
What this means in practice
- •For machine translation developers: Identify specific error types in machine-translated text by analyzing reconstruction difficulty at various masking levels.
- •For natural language generation engineers: Diagnose weaknesses in text generation systems by evaluating detailed performance signals from masking-rate conditioned scores.
Authors
Yiqi Liu, Kun Zhao, Chenghua Lin, Nafise Sadat Moosavi
Abstract
A single evaluation score can rank generation systems but rarely reveals what failed. We propose rate-resolved text evaluation, which turns the masking rate of masked diffusion language models into a diagnostic axis. By reconstructing text under different fractions of masked tokens, one diffusion score becomes a vector of band-conditioned scores that reflect reconstruction difficulty at varying levels of visible context. On SummEval, preserving this rate structure yields complementary signal: the best masking-rate band improves fusion with autoregressive likelihood over a collapsed diffusion scalar, survives a sample-shuffle control that breaks rate ordering, and adds residual signal to a fixed prompted judge. A second diffusion model reproduces the effect more weakly. On WMT MQM zh-en, roundtrip mask-and-reconstruct probing produces a tunable localization curve in which ranking-oriented AUROC and span-detection F1 peak at different masking rates. Factuality benchmarks bound the interpretation: band semantics are model- and task-dependent, so each application requires empirical validation.