SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created SteelBench, a new benchmark dataset to test vision-language models in real industrial CCTV footage, where workers are hard to see due to dust, low light, and other challenges. They carefully annotated over 1,300 video clips with detailed actions, safety rules, and protective gear information. Their work also addresses how human and model-assisted labeling can affect model evaluation by introducing an audit protocol. Testing nine models showed that even the best perform much worse than humans and struggle with safety rule understanding. SteelBench is made available for further research on Hugging Face.
vision-language modelsaction recognitionindustrial CCTVsafety-rule reasoningannotation provenancetemporal deduplicationPPE attributesmodel audit protocoldataset benchmarkmachine learning evaluation
Authors
Suryanarayana Reddy Yarrabothula, Manisha Chawla, Kunal Sinha, Gagan Raj Gupta, Sashank Lekkala, Ashirvadhan Dosapati, Saikamal Nannuri, Katragadda Ajay RamaSwamy Chowdary Gowtham
Abstract
Existing video benchmarks evaluate action recognition on consumer videos, egocentric recordings, or simulated industrial environments. They do not test vision-language models under the visual and procedural conditions of real industrial CCTV, where workers appear as distant figures amid dust, steam, low light, glare, occlusion, and overlapping activities. We introduce STEELBENCH, a diagnostic benchmark for industrial surveillance that jointly evaluates per-worker activity recognition, safety-rule reasoning, and annotation provenance. SteelBench contains 1,345 densely annotated clips, curated from 149 hours of operational plant footage and 10,024 candidate clips using temporal deduplication, class balancing, and visibility-aware stratified sampling. Each clip includes dense per-worker action labels, PPE attributes, spatial context, and safety-rule annotations. Because model-assisted annotation can shape the labels later used for model evaluation, SteelBench includes a provenance-aware audit protocol. The protocol measures label influence, evaluates sensitivity to ground-truth provenance, and reports a human reference from expert-reviewed labels. Applying this audit, we find that unaudited VLM-sourced ground truth can inflate same-family model accuracy by up to 17 percentage points. Across nine VLMs from four architectural families, the best model reaches only 42.6% action accuracy, compared with an 84.6% human benchmark. Performance also fragments across recognition, robustness, calibration, and safety reasoning. Even when models predict the correct action, 37-58% of cases still yield incorrect safety judgments, and no model passes more than 2 of 5 diagnostic checks. The dataset is publicly available on Hugging Face.