Fidelity-Diversity Metrics for Text
2026-07-06 • Computation and Language
Computation and Language
AI summaryⓘ
The authors worked on how to better judge the quality of extra text added to datasets that train language models. They created two new measures: one to check if the new text is similar to the original data (fidelity) and another to see if it covers all different types of content in the original data (diversity). Using these measures, they showed that problems in generated text come from either not matching the original content closely or not being varied enough. They also found that low diversity in synthetic math problem datasets leads to worse performance when models are trained on that data.
language modelingdata augmentationfidelitydiversityoptimal transportM2D2 datasetGSM8K datasetsynthetic datamodel evaluationprecision and recall
Authors
Amanda Wang, Tudor Manole, Florentina Bunea, John Thickstun
Abstract
As language modeling technology matures, there is an increasing research focus on the composition and curation of datasets used to train these models. For instance, practitioners commonly seek to augment high-quality datasets with additional text to enhance the performance of models trained on that data. However, informed decisions about data augmentation require more nuanced assessments about data quality. We build on work measuring the precision and recall of generative models to develop a pair of metrics that quantify (1) fidelity, capturing how closely candidate text resembles reference data, and (2) diversity, capturing how well it covers the modes of the reference dataset. Our metrics are based on optimal transport divergence functionals between discrete text summaries. In experiments on M2D2 text datasets, we show that these metrics are able to disentangle a lack of fidelity from a lack of diversity in deficient candidate text. In further experiments, our metrics detect diversity deficits in synthetic GSM8K-style math datasets, which correlate with degradations in downstream accuracy of language models finetuned on this synthetic data.