ProtoFusion uses protocols to improve infrared and visible image fusion
ProtoFusion: Protocol-Guided Prompt Refinement for Generative Infrared and Visible Image Fusion
Computer Vision and Pattern Recognition
Summary
Combining images from infrared and visible cameras can help us see more details in a scene. The authors developed a new method called ProtoFusion that guides this combination process using detailed instructions, called protocols, rather than fixed rules. These protocols specify what information should be kept or hidden and what qualities the final image should have. The method also learns to refine these protocols over time to produce better results, and it carefully fixes inconsistencies afterward. Their tests show that ProtoFusion creates clearer and more visually appealing fused images compared to existing methods.
What this means in practice
- •For security camera operators: Produce clearer combined infrared and visible security footage that highlights important scene details for better monitoring.
- •For remote sensing teams: Generate fused images from infrared and visible sensors to improve environmental or agricultural monitoring accuracy.
Authors
Pengwei Liang, Junjun Jiang, Junhui Hou
Abstract
Infrared and visible image fusion integrates complementary information from both modalities to provide a more comprehensive representation of the scene. We propose ProtoFusion, a protocol-guided generative framework for infrared and visible image fusion. Instead of relying on fixed fusion operators, hand-crafted objectives, or manually specified one-shot prompts, ProtoFusion formulates fusion as protocol-constrained generation. It represents fusion intent with a structured protocol, consisting of environment state cards, grounding principles, and a style profile, which explicitly specifies which evidence should be preserved or suppressed and which properties the fused result should satisfy. To optimize this protocol, we introduce protocol-guided prompt refinement, a closed-loop learning framework that iteratively performs generation, evaluation, and revision under multimodal evidence. Once learned, the optimized protocol is fixed at inference time and directly guides fusion generation. To address residual source inconsistencies, we further incorporate structured belief construction, which organizes modality reliability, scene evidence, and cross-modal conflicts into a structured representation for conservative correction after generation. Extensive experiments on standard benchmarks show that ProtoFusion improves quantitative performance, achieves favorable results in aesthetic evaluation, and produces visually coherent fusion outputs. Project Page: {https://erfect2020.github.io/ProtoFusion/}.