Multimodal AI struggles to recognize fire and smoke in safety tests
SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Understanding fire and smoke quickly and accurately is very important for keeping people safe during emergencies. The authors created a big test called SAFIRE, which has lots of pictures and questions about fire scenes to see how good AI models are at understanding them. They found that many AI systems still make mistakes and don’t recognize fire situations well. However, by training these systems with a small amount of special fire-related data, their accuracy improved a lot. This shows that carefully chosen examples can help AI do better in important safety tasks.
Multimodal large language modelsFire and smoke detectionSafety-critical tasksVision-language benchmarksMultiple-choice visual question answeringData annotation qualityVision encoder adaptationDomain-specific training dataAI accuracy evaluationDisaster response technology
Authors
Pengfei Li, Naufal Suryanto, Sicheng Zhang, Mohammad Alsharid, Muzammal Naseer
Abstract
Multimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation. We introduce SAFIRE, a large-scale benchmark for fire-smoke understanding in MLLMs, comprising 83K captioned images from 20 scenarios and 193K multiple-choice VQA (MCVQA) generated from a 9.7K-image subset, spanning 10 evaluation dimensions from basic perception to higher-order reasoning. A GPT-5.4-assisted multi-stage verification pipeline with MLLM majority voting ensures annotation quality. Evaluating ten open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing major gaps in safety-critical reasoning. We further show that adapting vision encoders with only 7% of our domain-specific data boosts fire-scene classification accuracy from 20.1% to 64.5%, indicating that carefully curated data can yield substantial gains even when data volume is limited. All datasets, models, and code are available at https://risys-lab.github.io/SAFIRE/.