Summary
Fusing infrared and visible images can help computers see better by combining different types of information, but existing methods usually only work well for tasks they have seen before. The authors propose a new approach called CRT-HMAR that adapts to new, unseen tasks by understanding which parts of the image matter for each task and balancing multiple tasks at once. Their system uses several cooperating agents to guide the fusion process based on what each task requires and to handle conflicts when optimizing for several tasks. Experiments show their approach works better on new tasks while still doing well on those it was trained for.
What this means in practice
- •For surveillance system engineers: Create infrared-visible fusion systems that adapt to new monitoring tasks without retraining for each specific scenario.
- •For autonomous vehicle developers: Implement fusion algorithms that dynamically adjust to different vision tasks like obstacle detection or navigation in mixed lighting conditions.
Authors
Zengyi Yang, Shuai Yuan, Zhong-Cheng Wu, Juan Cheng, Huafeng Li, Yu Liu
Abstract
Infrared and visible (IR-VIS) image fusion integrates complementary multimodal information into a single fused image to support downstream vision tasks. However, existing methods are typically tailored to seen tasks within a fixed task set and struggle to generalize to unseen tasks, which restricts their applicability in real-world open-task scenarios. To address this issue, this paper proposes CRT-HMAR, a Causal Requirement Tracing-Guided Hierarchical Multi-Agent Regulation Framework for open-task-aware IR-VIS image fusion. CRT-HMAR introduces a Causal Requirement Tracing Task Localization mechanism, which actively intervenes in key image information and observes task-network response variations to map task-specific semantic preferences into image-level causal requirement maps. Based on these maps, a requirement analysis agent aggregates task-specific requirement knowledge to adaptively guide requirement-customized image fusion. Moreover, CRT-HMAR incorporates History-Analysis Multi-Objective Balancing and Task-Level-Correction Conflict Mitigation mechanisms, jointly constructing a hierarchical regulation chain of "requirement interpretation - task balancing - conflict mitigation". Through multiple collaborative agents, CRT-HMAR dynamically regulates key processes including open-task requirement modeling, multi-task balanced optimization, and gradient conflict mitigation. Extensive experiments on open-task scenarios involving five downstream tasks demonstrate that CRT-HMAR significantly improves generalization to unseen tasks while maintaining the performance and balance of seen tasks. Overall, CRT-HMAR shifts IR-VIS image fusion from task-oriented modeling toward requirement-oriented modeling, promoting its extension from closed-task settings to real-world open-task scenarios.