Graph memory helps multiple agents solve complex multimodal problems
GraMRAG: Orchestrating Multi-Agent Multi-Step Reasoning via Graph Memory with Reinforcement Learning
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Answering complex questions that need many steps and combine images and text can be hard because machines forget what they saw or looked up earlier. The authors propose a new way to help multiple 'agents' share knowledge using a graph that records their actions and observations, so they don’t repeat work or get confused. They also teach the agents to focus on the most important parts of this graph, improving their reasoning across multiple steps. Tests show this method works better than older ones on hard tasks involving images and text.
What this means in practice
- •For multimodal ai developers: Build AI systems that combine images and text to answer complex questions requiring many reasoning steps using graph-based memory and learning.
- •For interactive ai system builders: Create systems that track reasoning progress explicitly to avoid redundant data lookup and improve decision making in long conversations.
Authors
Zhongyu Wang
Abstract
Although existing multi-agent Retrieval-Augmented Generation (RAG) systems have demonstrated promise on complex multimodal reasoning tasks, they remain fundamentally limited in reasoning depth and memory structure, suffering from inadequate retrieval and state blindness when answering knowledge-intensive questions. To address these limitations, we propose GraMRAG, a graph memory-guided multi-agent RAG framework that integrates a dynamic multimodal memory graph to enable stable, multi-step multimodal reasoning. We introduce a vision-text bridged reasoning paradigm that unifies multi-scale entity cropping with a ReAct-style visual toolchain, enhancing the long-horizon cross-modal reasoning capability. We further construct a multimodal memory graph that formalizes agent reasoning as a dynamic directed acyclic graph (DAG), explicitly modeling action-observation dependencies to mitigate state blindness and suppress redundant retrieval. Moreover, we propose Topology-Aware Policy Optimization (TAPO) that leverages graph topology for critical path identification and targeted node pruning, enabling fine-grained credit assignment across multi-step reasoning trajectories. Extensive experiments on challenging multimodal benchmarks demonstrate that our approach consistently outperforms existing baselines and achieves state-of-the-art performance on complex long-horizon reasoning tasks.