Large vision language models fall short in spotting warehouse hazards
LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Identifying safety hazards in busy warehouses is tough for computer models because there isn’t enough real-world data to train them well. To help, the authors collected thousands of images and videos from actual warehouses, along with many questions and answers about potential dangers involved. They tested popular advanced vision-language models on this data and found that even top models struggle compared to human experts. This shows there’s still work to do before these models can reliably help keep industrial workplaces safe.
What this means in practice
- •For industrial safety teams: Use the LogiScope-VQA benchmark to assess and improve AI models for recognizing hazards in real warehouse environments.
- •For logistics operators: Train custom AI systems with LogiScope-VQA data to enhance automated monitoring and risk detection in logistics parks.
Authors
Hanjing Zhou, Mingze Yin, Ying Lian, Jun Ma, Chang-Yu Hsieh, Yanbing Zhou
Abstract
Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incorporate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identification poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMs' practical deployment in real-world settings. The industrial dataset is publicly available under the CC BY-NC-SA 4.0 license.