Monarch MoE reduces forgetting in multimodal AI on limited memory

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Machine Learning

Summary

Multimodal AI systems learn from different types of data like images and text but often forget earlier tasks when learning new ones. The authors introduce Monarch-MoE, a new method that divides learning into smaller, separate parts to prevent forgetting while keeping all parts connected. This method is designed to work well even on computers with limited memory, like a single 24 GB graphics card. Their tests show Monarch-MoE retains earlier knowledge much better and uses fewer resources.

What this means in practice

  • For ai system engineers: Enable efficient adaptation of multimodal models on hardware-limited devices by reducing forgetting and memory use during continual learning.
  • For edge computing developers: Implement continual instruction tuning in vision and language applications on single GPU systems with constrained memory.

Authors

Xi Zhang, Boyu Diao, Zesen Chen, Ruiqi Liu, Hangda Liu, Yongjun Xu

Abstract

Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. This structure provides blockwise gradient isolation: individual block-pair updates affect only one output block, while connections between all input and output blocks are maintained. To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass. Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively. In the main comparison, first-task forgetting is as low as 0.52 percentage points on MLLM-DCL and 0.10 on UCIT, with method-dependent trade-offs in final performance. Our implementation supports fine-tuning and inference on a single 24 GB RTX 4090, with lower measured memory use and competitive runtime in the evaluated settings. Together, these algorithmic and systems improvements make structured experts a practical option for continual adaptation on memory-limited hardware.