Vision language models match human accuracy on video annotation tasks
Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Watching videos to understand human behavior usually needs people to carefully label what they see, which takes a lot of work. The authors studied how well vision-language models (VLMs) can do this job by themselves and how they can help people. They found that these models almost match humans in accuracy when labeling videos and that humans checking the models’ work do even better while saving time and money. Their work shows the right ways humans and AI can work together to analyze videos about people efficiently.
What this means in practice
- •For video annotation teams: Automate human-focused video labeling with vision-language models to achieve near-human accuracy and reduce labeling effort.
- •For content moderation teams: Improve video review accuracy and efficiency by combining vision-language models with human verification in annotation workflows.
Authors
Xiyuan Shen, Jiuyang Lyu, Seokhyun Hwang, Huanfen Yao, Shwetak Patel, Zhihan Zhang, Jacob O. Wobbrock
Abstract
Video provides a rich record of human behavior, interaction, and situated contexts, offering important evidence for understanding people and conducting human-centered research. As vision-language models (VLMs) become increasingly capable of analyzing video, they offer opportunities to automate this traditionally human-intensive process. Yet a central question remains: when can VLMs analyze human-centered video independently, and when does reliable analysis still require human involvement? To address this question, we first characterize video analysis practices in human-centered research. We systematically analyze all 1,702 CHI 2026 full papers and identify 125 that annotate videos. Through iterative coding, we derive a five-dimensional taxonomy spanning analytic purpose, viewpoint, phenomenon, reasoning requirement, and annotation authority. Grounded in recurring annotation tasks captured by this taxonomy, we construct a benchmark of 15 representative tasks from open datasets to map the capabilities and limitations of a general-purpose VLM. We examine the division of labor between humans and VLMs by comparing three annotation workflows: VLM alone, human alone, and human verification of VLM outputs. Across tasks, VLM-alone annotation approaches human accuracy on average (HNS = 97.0, where 100 denotes human-alone performance), demonstrating substantial potential to automate human-centered video analysis. Human verification achieves the highest accuracy (HNS = 121.5) while reducing human annotation time by 48.9% and monetary cost by 31.3%-44.5% relative to human-alone annotation. Our findings connect real-world human-centered video analysis tasks and current VLM capabilities, and clarify how human-AI collaboration can make VLM-assisted analysis reliable and efficient.