Framework evaluates recaptioned image text quality under fixed budget
A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributions
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Training text-to-image models often uses large collections of images paired with descriptive captions. The authors identify that evaluating these collections for quality and content is difficult because the captions come from different sources and processes. They propose a new method that measures how good and faithful caption descriptions are across multiple dimensions while keeping the amount of text fixed. Their method helps compare different captioning approaches and datasets in a consistent way. They also release a large audited dataset for others to use.
What this means in practice
- •For machine learning engineers: Evaluate and compare large image-text datasets systematically during model training to ensure quality captions under fixed text limits.
- •For data curation teams: Audit and select the best recaptioned corpora to improve training data quality for text-to-image generation projects.
Authors
Giyeong Oh, Junghun Park, Yuhan Bae, Youngjae Yu
Abstract
Recaptioned image-text corpora are now standard for text-to-image (T2I) training, with vision--language model (VLM) captioners replacing sparse alt-text by dense descriptions. A recaptioned corpus is a supervision distribution induced by a documented captioning policy ($π$), captioner ($V_c$), and source corpus ($C$). Length-correlated proxies miss caption-register artifacts and downstream T2I benchmarks entangle the corpus with training choices, so this distribution is hard to audit at corpus scale. We introduce a reusable matched-budget audit framework for recaptioned supervision distributions $D_{π,V_c,C}$: at a fixed text budget of $B = 64$ it reports a five-axis profile spanning prompt-side coverage, image-conditioned faithfulness, and caption-surface health, with claimed controllable basic units (CBU) as the common claim unit. We instantiate the framework on seven paired comparisons over five public source corpora. Across the four cross-corpus pairs, the released surface raises supported CBU per caption by $+3.39$ to $+6.36$ under both Qwen and Gemma Judges, and on CC12M the same framework exposes a long-vs-dense frontier that is consistent across both judges and four budgets. We release the audited multi-source recap corpus ($\approx$ 490M) together with the audit-artifact bundle.