Visually grounded captions improve ai generated video quality assessment
Beyond Source Prompts: Towards Visually Grounded AI-Generated Video Quality Assessment
Computer Vision and Pattern Recognition
Summary
Videos made by artificial intelligence often have flaws that can be hard to spot or judge using simple text descriptions alone. The authors propose a new way to check video quality by using captions that are created by looking closely at the video itself, not just the original text prompt. These captions describe the whole video and smaller parts of it, helping to better understand how good the visuals and timing are, and how well the video matches the text. Tests show their method works better than previous ones for judging AI video quality.
What this means in practice
- •For video production teams: Provide more accurate quality checks for AI-generated videos by analyzing video content with updated captions that reflect the actual visual output.
- •For content moderation teams: Improve evaluation of generated video quality and relevance to prompts, aiding decisions on video suitability for publishing or distribution.
Authors
Kehao Zhou, Yixuan Li, Yuxuan Jiang, Fan Zhang
Abstract
Recent advances in generative artificial intelligence have accelerated video generation technologies. However, AI-generated videos (AIGVs) remain susceptible to various distortions, highlighting the need for dedicated video quality assessment methods. Content semantics play an important role in AIGV quality assessment, with source prompts typically used as textual references. However, these prompts may not accurately reflect the generated video content, resulting in a discrepancy between the intended textual description and the generated result. To address this limitation, this paper proposes VGCap, an AIGV quality assessment framework guided by visually grounded captions that align better with generated content. Specifically, we introduce two forms of captions: a global caption obtained by minimally revising the source prompt based on multi-frame visual evidence, and prompt-independent segment captions that describe localized temporal content. Together, these captions provide complementary textual evidence for assessing spatial quality, temporal quality, and text-video alignment. Experimental results on the LGVQ and FETV datasets show that VGCap outperforms representative baselines across all three dimensions. Further ablation studies on caption sources and temporal components confirm the effectiveness of visually grounded captions for each quality dimension, while highlighting the benefits of segment-level features for temporal quality prediction. The source of VGCap will be available at https://github.com/lolo123-123/VGCap.