Multimodal models tested on detailed fire and smoke safety tasks
SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Understanding fire and smoke from images is important for keeping people safe during disasters. The authors created a big collection of fire and smoke pictures with captions and questions to test how well AI models can understand them. They found that current models struggle with safety reasoning in these situations but improved a lot when trained on specific fire-scene data. This shows that having focused, high-quality data helps AI better handle important safety tasks.
What this means in practice
- •For emergency response teams: Improve AI tools for recognizing fire and smoke in varied real-world images to support faster disaster assessment and response decisions.
- •For autonomous vehicle developers: Enhance multimodal perception systems to better detect fire hazards on roads and improve vehicle safety protocols in dangerous environments.
Authors
Pengfei Li, Naufal Suryanto, Sicheng Zhang, Mohammad Alsharid, Muzammal Naseer
Abstract
Multimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation. We introduce SAFIRE, a large-scale benchmark for fire-smoke understanding in MLLMs, comprising 83K captioned images from 20 scenarios and 193K multiple-choice VQA (MCVQA) generated from a 9.7K-image subset, spanning 10 evaluation dimensions from basic perception to higher-order reasoning. A GPT-5.4-assisted multi-stage verification pipeline with MLLM majority voting ensures annotation quality. Evaluating ten open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing major gaps in safety-critical reasoning. We further show that adapting vision encoders with only 7% of our domain-specific data boosts fire-scene classification accuracy from 20.1% to 64.5%, indicating that carefully curated data can yield substantial gains even when data volume is limited. All datasets, models, and code are available at https://risys-lab.github.io/SAFIRE/.