Code captions improve video reasoning and reconstruction accuracy
CodeCap-Bench: Can Code Serve as a Good Video Caption?
Computer Vision and Pattern Recognition
Summary
Describing videos with words can be slow and unclear when showing detailed movements and layouts. The paper explores using short computer programs, or code, to describe videos instead of just natural language. The authors found that code descriptions help computers understand video content better for answering questions and recreating the video. However, current AI models still have trouble writing accurate code for complex movements. They created a new test called CodeCap-Bench to measure how well AI creates these code captions.
What this means in practice
- •For video analytics teams: Improve video content understanding by using executable code captions to enable more accurate visual reasoning and question answering.
- •For multimodal ai developers: Develop better multimodal models that generate faithful code representations of videos to enhance multimedia understanding.
Authors
Jiafu Tang, Yifan Xu, Eric Tang, Junjie Zhao, Shipeng Luo, Te Sun, Haoxuan Hu, Yiyan Ji, Xinlong Chen, Zihao Wang, Yang Shi, Yuanxing Zhang, Pengfei Wan, Jiaheng Liu
Abstract
Video captioning aims to preserve visual information through compact descriptions, predominantly using natural language. However, precisely expressing fine-grained spatial structures and temporal dynamics in natural language often leads to lengthy and potentially ambiguous descriptions. In this work, we explore Code as Video Caption, a paradigm where executable programs serve as a complementary representation of visual content through graphical primitives, layout rules, and functions of time. To examine whether code captions effectively preserve video information, we compare them with natural language captions on downstream question answering and video reconstruction tasks. Our results show that code captions achieve superior reasoning accuracy and reconstruction fidelity. Despite these advantages, realizing the potential of code captioning depends on whether current multimodal large language models (MLLMs) can generate faithful code captions. To systematically assess this capability, we introduce CodeCap-Bench, which comprises 323 videos spanning five categories of structured 2D content, along with fine-grained annotations and a multi-level evaluation framework. Extensive experiments reveal that while leading proprietary MLLMs demonstrate promising code captioning capabilities, they still struggle to faithfully capture complex temporal dynamics, with open-source models lagging substantially behind.