Adaptive interaction improves multimodal agent debate accuracy and efficiency

AIM: Adaptive Interaction in Multi-Agent Debate for Multimodal LLM Inference

Multiagent Systems

Summary

Multimodal tasks use different types of information like text, images, and audio to solve problems, but current ways for agents to talk to each other rely on just one fixed method, which can miss details or waste effort. The authors propose AIM, a system that lets agents choose the best type of interaction—like sharing specific image parts or audio clips—depending on what the problem needs. This approach helps agents work together more precisely and efficiently, improving accuracy while using fewer words. AIM adapts how agents share information for each question, avoiding unnecessary or missing evidence.

What this means in practice

  • For multimodal ai developers: Improve accuracy and token efficiency in AI systems by adapting interaction methods based on task-specific multimodal evidence during agent debate.
  • For chatbot platform engineers: Design chatbots that dynamically select communication modalities like text or image regions among AI agents to provide better and faster reasoning on complex queries.

Authors

Wei Fan, JinYi Yoon, Bo Ji

Abstract

Multi-Agent Debate (MAD) improves reasoning on multimodal tasks, but its accuracy and token efficiency depend on how agents interact during debate. A largely overlooked design choice is which interaction modalities agents use to exchange evidence. Existing methods rely on fixed text- or graph-based interaction, which fails to convey fine-grained multimodal evidence and thus degrades accuracy while inflating token cost due to verbose descriptions. To address this, we propose Adaptive Interaction in Multi-Agent Debate (AIM), an adaptive routing framework that dynamically selects the optimal modality combination for inter-agent interaction in multimodal MAD. Beyond text and graph, AIM introduces task-dependent modality interaction, where agents exchange targeted regions of interest from the task-specific multimodal input, such as image regions, audio segments, or spatio-temporal video clips. To determine the best combination for each instance, AIM first generates a structured baseline response and extracts interpretable routing features that capture which evidence the instance requires, how confident the baseline agent is, and how much new evidence each modality would add beyond the baseline response. A lightweight router then uses these routing features to select a modality combination that avoids both under-fusion (i.e., agents miss decisive evidence) and over-fusion (i.e., redundant modalities inflate token cost) risks. Across eight benchmarks spanning three multimodal task families, \name achieves the highest accuracy. Combining modalities improves accuracy by up to 6.4\% over the best single-modality baseline, and adapting the combination per instance adds up to 4.2\% over the most accurate fixed combination of each benchmark with up to 53.4\% fewer tokens.