Vision-language models reduce video annotation costs with simple heuristics
Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics
Computer Vision and Pattern RecognitionComputation and LanguageSocial and Information Networks
Summary
Annotating videos using vision-language models can be very expensive because they process a lot of data. The authors studied simple ways to save money, like looking at fewer video frames or using only parts of the information. They found that some cheaper methods still give good understanding without misleading results, especially using small image grids instead of full videos. This helps people who need video insights get reliable results while paying less.
What this means in practice
- •For computational social science teams: Annotate short videos efficiently by using shot-transition based image grids to cut token costs while preserving analysis accuracy and validity.
- •For social media analytics teams: Reduce video content processing costs by selectively sampling frames or using single modalities without losing key insights on sentiment and topics.
Authors
Zhixi Zhu, Kristina Gligoric
Abstract
Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one frame per second requires millions of tokens. To reduce costs, researchers rely on heuristics such as sampling a subset of frames, compressing videos into image grids, or using only a single modality. However, it remains unclear which heuristics save cost, and whether they preserve the downstream conclusions these annotations enable. To address this gap, we conduct a systematic evaluation of these heuristics using short-form videos, on two computational social science (CSS) tasks: sentiment and topic classification. We evaluate each configuration along three axes the literature typically treats separately: classification accuracy, validity of downstream inference, and per-video token cost. First, we find that accuracy and validity diverge: the highest-accuracy configuration can produce wrong conclusions. Second, modality value is not guaranteed: text alone can yield strong performance, indicating that adding modalities can add cost without adding signal. Finally, we find that cost can be decoupled from video length when annotating short-form videos: a single $2\times8$ image grid built via simple shot-transition detection approaches full-video understanding ($κ$ within~.05), at $\sim 15\%$ of the token cost. Based on these findings, we derive guidelines that can enable cost-aware VLM annotation in CSS.