Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present DG-Mem, a memory system designed to help large multimodal language models (MLLMs) improve scientific and mathematical reasoning without changing their core parameters. DG-Mem adds an external memory divided into specific examples (exemplars) and general rules (schemas) created from abstract reflections, inspired by how human memory works. It uses an online process to organize concepts dynamically and a method to measure how useful each rule is, adjusting what it recalls during testing. This approach works without retraining the model and improves performance on challenging benchmarks across various model types.
multimodal large language modelsnon-parametric memoryComplementary Learning Systemsexemplar memoryschema memoryonline concept categorizationShapley valueparameter freezingmathematical reasoningmodel adaptation
Authors
Jieke Wang, Tiancheng Shen, Yibo Yang, Ming-Hsuan Yang
Abstract
Frontier multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning. Parameter-level adaptation is unavailable for closed-weight or on-device backbones, and stateless prompting forfeits any compounding benefit from problems already solved. We propose \textbf{DG-Mem}, a dual-grained agentic memory framework that augments a frozen MLLM with a non-parametric, externally stored memory built once from training-time rollouts and consulted read-only at test time. Motivated by the Complementary Learning Systems (CLS) account of human memory, DG-Mem factors its store into an instance-grounded exemplar memory and a category-level schema memory of IF-THEN rules, with a transient reflection store mediating their construction so that schemas are synthesized only from abstract reflections, never from exemplar text. Two design choices distinguish DG-Mem: an online concept categorizer that grows the category space incrementally during training rather than committing to a predefined taxonomy, and a Shapley context attribution procedure that decomposes correctness across the entire retrieved rule set and yields a per-rule utility that re-weights retrieval at test time. The pipeline introduces no gradient updates and is deployable on closed-weight or on-device backbones. Across MathVista, MMMU, and MMMU-Pro on four open-weight and proprietary backbones (Qwen3.5-27B, Qwen3.5-122B-A10B, GPT-5-Nano, Gemini-3-Flash), DG-Mem improves consistently over no-memory and competitive memory baselines.