Video caption scores often match but fail in opposite ways
Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation
Computer Vision and Pattern RecognitionComputation and Language
Summary
Existing methods to check how good video captions are often borrow tools made for single pictures, but these tools don't reflect what people actually think about captions for videos. The authors found that while automatic scores might look similar, the actual mistakes in captions differ a lot and aren’t captured by these scores. They also discovered that looking at captions in small parts reveals more errors than treating the caption as a whole. The authors suggest that evaluation should focus on what kind of mistakes happen, not just how often errors occur.
What this means in practice
- •For video processing teams: Improve video caption quality checks by using detailed failure categories instead of single summary scores.
- •For machine learning engineers: Develop evaluation tools that identify specific types of video caption errors to guide model improvements.
Authors
Ankit Khedia, Astitva Jaiswal, Chaitanya Prasad, Anisha Raju, Aangeeras
Abstract
Video captioning models are increasingly evaluated with metrics designed for single-sentence image captions. We test whether those metrics recover human judgement on dense, multi-window video description, and find that they do not. Six frontier models captioned 196 short clips. Trained annotators post-edited every caption against a failure taxonomy fixed before annotation, once as a whole-clip overview and once window by window, and a quality-control reviewer checked each annotation, yielding 1,144 overview verdicts and 4,672 window verdicts. Reference-overlap metrics do not separate the captions reviewers accepted from those they rejected: BLEU-4 reaches an AUC of 0.510 and a CIDEr-style tf-idf cosine 0.528, and restricting references to reviewer corrections leaves BLEU-4 at 0.479 (95% CI 0.431-0.529), indistinguishable from chance. Review granularity changes what is detected. The same annotators record camera and shot errors on 2.7% of captions when reading the overview and on 12.8% when reading window by window, a 4.7x gap that holds across every robustness check we ran, on the mode most likely to need a near-total rewrite. Correct-overview rates span 54.1-72.7%, but only 4 of 15 pairwise gaps survive Bonferroni correction (5 after adjusting for reviewer and clip effects), and accuracy and derived severity order the models differently. Models with statistically indistinguishable accuracy have sharply different failure profiles, which no single scalar shows. We argue that caption evaluation should record what was wrong rather than only how often, and we report the taxonomy, the per-model profiles, the corpus properties that predict difficulty, and the code that regenerates every number in the paper.