Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation
2026-06-22 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputer Science and Game Theory
AI summaryⓘ
The authors study how different vision-language models (VLMs) judge physical correctness in videos. Instead of using one set of rules for all models, they create a special rulebook tailored to each model's unique way of understanding physics. They do this by asking each model to list physical errors, then improving these lists using human feedback and large language models. Their method, called JudgeFit, works better than a one-size-fits-all approach and reveals the unique weaknesses of each model.
vision-language modelsvideo generationphysical consistencyevaluation taxonomylarge language modelsmodel calibrationcommonsense reasoningbenchmarkingiterative refinement
Authors
Yu Cao, Ziquan Liu, Zhensong Zhang, Jiankang Deng, Shaogang Gong, Jifei Song
Abstract
Maintaining physical consistency in video generators and world models increasingly relies on vision-language models (VLMs) as automated judges that provide reward signals, ranking decisions, and data-filtering criteria. Yet VLMs differ substantially in training data and architecture, encoding physical phenomena through distinct internal representations. A single global evaluation schema therefore gives every VLM the same axes of competence, regardless of what each can actually perceive. We propose JudgeFit, an iterative refinement procedure that discovers a per-VLM evaluation taxonomy. An initial taxonomy is constructed by prompting the target VLM to enumerate physics errors on a small set of videos and clustering the resulting descriptions. The taxonomy is then refined through a diagnostic step: we calibrate the VLM's per-dimension scores to human physical-commonsense ratings, diagnose which dimensions it scores unreliably or redundantly, and prompt an LLM to repair them, iterating until convergence. We further instantiate this procedure as a benchmark and apply it to 16 VLMs spanning eight model families. The refined taxonomy outperforms the global-schema baseline on held-out videos for every VLM tested, with a mean relative improvement of approximately 32%. Beyond aggregate accuracy, the per-VLM profiles expose model-specific blind spots that overall rankings cannot anticipate, with reliability patterns differing markedly across model families.