CAPEval: A Decoupled Caption Evaluation across Understanding and Generation
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new way to evaluate captions that describe images by separating two important aspects: how much of the image’s details the caption covers (Coverage) and how accurate the caption’s claims are (Precision). They tested 10 different captioning methods and found that depending on the task, either thoroughness (Coverage) or accuracy (Precision) matters more. This new evaluation helps better understand caption quality and guides choosing or improving captioning models for specific uses. Essentially, it breaks down caption quality instead of using just one overall score.
captioningmultimodal understandingtext-to-image generationcaption qualityCoveragePrecisionevaluation benchmarkground-truth captionsmodel familiesdownstream tasks
Authors
Zhipeng Liu, Haochen Wang, Zhaoxiang Zhang
Abstract
Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.