Video understanding method cuts computing by using captions and few frames
Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Watching and understanding long videos on small devices is hard because there is limited power and internet speed. The authors found that text summaries of videos keep the story better over time, but pictures are important for details like what things look like. They made a system that makes one detailed caption of the whole video first and then, when you ask a question, it decides if it really needs to look at pictures to answer or if the text is enough. This saves a lot of computer work and still gives good answers about the video. Their tests showed it works well on long videos while using much less computing power.
long-video understandingedge computingcaptioningvisual tokenslanguage modelsquery-based retrievalvisual-text dualitykeyframe extractionmulti-modal learningcost-efficient processing
Authors
Weitong Cai, Hang Zhang, Yukai Huang, Yiqiao Xie, Shan Gao, Jiankang Deng, Songcen Xu, Jifei Song, Zhensong Zhang
Abstract
Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets. Subsampling visual tokens loses temporal structure, while text-only video memories lose fine-grained visual attributes. We observe a visual-textual duality: language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception. Building on this insight, we propose Caption-once, Frames-onDemand (CFD), a budget-aware edge-cloud agentic framework. The edge runs a single offline captioning pass that builds a dual-track narrative index, an event-level story skeleton plus a clip-level micro-log, cached and reused across queries without re-captioning. At query time, a cloud-side MLLM reasons over the index in a story-first loop centered on a lightweight Visual-Need Router: a per-query gating module that triggers bounded keyframe retrieval only for perceptual questions (appearance, on-screen text, attribute disambiguation) and keeps temporal-structural questions in language space. The router turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length. Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.