CapQuiz improves evaluation of video captions with multiple-choice tests
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Computer Vision and Pattern Recognition
Summary
Video captions are hard to judge because many good captions describe the same video differently. The authors created CapQuiz, a new way to evaluate captions by checking if they help answer multiple-choice questions about the video. These questions cover different types and topics, and do not rely on comparing captions to fixed references. This approach gives better agreement with human judgment and shows where captioning models do well or struggle.
What this means in practice
- •For ai developers: Test and improve video captioning models by automatically assessing captions through question answering without needing reference captions.
- •For content moderation teams: Evaluate video descriptions for accuracy and relevance to support quality control in user-generated content platforms.
Authors
Zizhen Wang, Bo Feng, Zhengfeng Lai, Shiyu Li, Yang Lu, Meng Cao, Ping Huang, Xiaoming Simon Wang
Abstract
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.