Vision language models lag in fixed camera infrastructure tasks
VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models work well when analyzing videos centered on people, but they struggle with videos from fixed cameras used in things like safety monitoring or logistics. The authors created VANTAGE-Bench, a new test set to measure this gap using videos and images from logistics, transportation, and smart spaces. They tested 17 models and found specific weaknesses in identifying events and tracking objects over time. Their benchmark also shows that some tasks related to spatial understanding are less affected, revealing where future improvements are most needed.
What this means in practice
- •For logistics operations teams: Assess and improve automated monitoring systems for inventory and package handling using fixed camera vision-language data.
- •For traffic management centers: Evaluate vision-language model capabilities for correctly identifying and tracking transport events on fixed infrastructure cameras.
Authors
Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain, Lap Fung Chan, John Suchanek, Yu Wang, Varun Praveen, Tomasz Kornuta, Vidya Nariyambut Murali
Abstract
As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric consumer video. This overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed cameras for open-loop insights like safety monitoring and operational logging. We introduce VANTAGE-Bench, a benchmark measuring this "Infrastructure AI Gap." It spans three operational domains (Logistics, Transportation, and Smart Spaces), unifies image and video evaluation across semantic, spatial, temporal, and spatio-temporal capabilities, and moves beyond multiple-choice to eight task formulations including dense captioning and spatio-temporal grounding. It adds a single-pass trajectory protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 media assets: 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes. Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated, not general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither scale nor proprietary access explains the pattern. Data, evaluation harness, and leaderboard: https://vantage-bench.org/