Video captioning system requests targeted observations for accuracy
OACap: Caption-Guided Active Observation for Detailed Video Captioning
Computer Vision and Pattern RecognitionComputation and LanguageMultiagent SystemsMultimedia
Summary
Creating detailed captions for videos is difficult because the system must decide what parts of the video to look at and how to put that information together. The authors propose a system called OACap that asks for specific new observations when it lacks details, then updates its description accordingly. They also built new tests to check how well captions keep facts and meaning intact. Their system improves caption quality by focusing on accurate and important video details.
What this means in practice
- •For video content creators: Generate more accurate and detailed captions that adaptively refine descriptions based on unseen video segments.
- •For multimedia indexing teams: Improve video indexing by using captioning that actively requests missing information for better metadata quality.
Authors
Hao Shi, Hong Yan, Xiaoyang Zhang, Qing Cui, Chao Huang
Abstract
Unlike question answering, where a query specifies a perceptual target, open-ended detailed captioning must determine what evidence to acquire and how to integrate it into a coherent global description. We propose Observation-Adaptive Video Captioner (OACap), a caption-centered agent that turns unsupported or missing information into targeted observation requests, uses spatiotemporal tools to acquire evidence, and iteratively verifies and revises the caption. OACap captions consistently improve performance across general video-understanding benchmarks using a fixed text-only reader. To better diagnose the information they preserve, we introduce Factual-Semantic Captioning Bench (FSCap-Bench). Its Core-Support Factuality Score (CSFS) combines claim precision with importance-aware fact coverage and responds as expected to four controlled perturbation types on 93.0-98.3% of valid caption pairs. Its Cross-Modal Semantic Round-Trip Score (CSRS) measures semantic preservation through a video-caption-video-caption round trip by matching atomic facts from source and recovered captions. Together, they connect downstream textual-memory utility with factual and semantic preservation.