Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation
2026-06-29 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created Rigel, a new way to automatically judge how good image and video captions are. Unlike older methods, Rigel uses a special part of a big language model trained to focus on judging captions without being distracted by complex language details. They improved Rigel further by training it with real human feedback using a new large video-caption dataset called Vid-Lepus. Tests showed Rigel does better than current popular methods, especially when it has to evaluate captions without any example references.
image captioningvideo captioningautomatic evaluationlarge language modelsself-distillationhuman judgment alignmentscoring metricsVid-Lepus datasetreference-free evaluationActivityNet-Fact
Authors
Shuitsu Koyama, Kazuki Matsuda, Yuiga Wada, Shinnosuke Hirano, Daichi Yashima, Komei Sugiura
Abstract
Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgments. Recent approaches using large language models (LLMs), commonly referred to as LLM-as-a-Judge, have improved alignment with human judgments but still suffer from a mismatch between large-vocabulary language modeling and evaluation over a small label set. To address this, we propose Rigel, an automatic evaluation metric for image and video captioning, based on self-distilled score adaptation. The metric employs an evaluation-specific scoring head distilled from a frozen LLM, which captures judgment signals in a task-aligned space without relying on large-vocabulary token sets. We then refine the LLM backbone with human judgment data. To train Rigel, we constructed the Vid-Lepus dataset, which contains 3,338 video clips, 33,380 reference captions, and 5,637 candidate captions. Experiments on multiple benchmarks show that Rigel outperforms state-of-the-art metrics, achieving over 10-point improvements on ActivityNet-Fact in the reference-free setting.