Multimodal models show varied accuracy on video analysis tasks
What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis
Computer Vision and Pattern RecognitionComputation and Language
Summary
When computers analyze videos to understand their content, they can look at the pictures, hear the sounds, or read text extracted from the video. This paper finds that no single input type—video, audio, or text—always works best for understanding videos. Combining these inputs does not consistently improve accuracy either. Even the best computer judgments can still differ quite a bit from human annotations, meaning it's important to carefully check how video understanding models are used depending on the task.
What this means in practice
- •For social science research teams: Choose and validate video input types carefully when using language models to analyze video content for social science studies.
- •For content moderation teams: Assess multiple input sources in video content analysis to improve detection accuracy and better match human judgments.
Authors
Christian Pipal, Eva-Maria Vogel, Morgan Wack
Abstract
Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input. We use the term representation validity to examine how text, audio, video, and combined inputs affect measurement and subsequent analysis. We compare input representations across six open-weight MLLMs and three human-annotated video datasets. No representation consistently performs best. On seven news-related constructs, native-video performance varies little across models (macro-F1 = .70-.77), while extracted-text performance ranges from .52 to .80. Three models perform better with text and three with video. Combining inputs rarely improves performance. Higher classification accuracy also does not reliably produce estimates closer to those based on human codes. The highest-F1 representation best reproduces human-coded regression coefficients in 17 of 42 comparisons. Even when each model uses its highest-F1 deployable representation, 45% of sample-prevalence estimates differ from the human-coded benchmark by more than ten percentage points. Input representation therefore requires validation for both the construct being measured and the quantity estimated from the resulting labels.