Simulation-based vision system improves wildfire monitoring and reporting accuracy
A Simulation-Grounded Agentic VLM Framework for Wildfire Monitoring and Reporting
Computer Vision and Pattern RecognitionGraphics
Summary
Wildfires are hard to track and understand because real video data with detailed fire behavior is rare and expensive to create. The authors developed a system that automatically turns wildfire simulations into videos with labels that describe fire and environment conditions. Their approach uses these videos as a memory to help a multi-agent model better identify and report wildfire attributes with higher accuracy than previous methods. They tested the system both on simulated data and limited real drone footage to show its effectiveness.
What this means in practice
- •For wildfire response teams: Generate detailed wildfire reports automatically from simulated and real UAV video to aid in timely situational awareness and decision making.
- •For drone operators: Improve interpretation of aerial wildfire videos by using a memory-augmented vision system that relates video cues to fire dynamics for better monitoring.
Authors
Duowen Chen, Yuchen Sun, Zhiqi Li, Yuxuan Liao, Sinan Wang, Bart van Bloemen Waanders, Bo Zhu
Abstract
Effective wildfire monitoring requires relating visual evidence to physical fire dynamics, yet real videos with synchronized physical annotations are scarce and high-fidelity 3D simulation is costly. We present a simulation-grounded vision-language model (VLM) framework that automatically converts 2D wildfire simulations into labeled video episodes. A fixed Blender mapping produces low-detail 3D proxies aligned with simulator terrain, fuel layout, fire activity, and wind cues; controllable video generation supplies richer appearance. The proxies are intermediate representations rather than finely rendered final scenes. Generated videos and simulator labels form reusable multimodal memory for a training-free multi-agent VLM system that retrieves reference episodes, reconciles visual and memory-based predictions, and produces structured wildfire reports. On held-out generated episodes, video memory achieves 51.5% exact four-tag accuracy, compared with 22.6% for direct VLM querying and 16-17% for text-only memory; the complete system achieves 77.3% accuracy on six simulator-derived report fields. Component ablations, cross-generator tests, and three real-UAV evaluations assess retrieval, reporting, generator changes, and observable monitoring tasks. The framework connects automatic simulation-to-proxy conversion with memory-based VLM reasoning under scarce real-world physical annotations.