Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents
2026-08-24 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors study how multimodal large language models (MLLMs), which use both language and visual information, can get overwhelmed by too much self-generated text during long tasks. They find that unnecessary reasoning text can hide important visual clues, so they create a method called SPARE to carefully remove this extra text without losing key visual info. SPARE uses a special way to check if removing parts of the text affects future reasoning by comparing the original and summarized contexts. Their method keeps the model accurate while cutting out a lot of redundant text, helping the model focus better on visual evidence.
Multimodal Large Language ModelsTask DecompositionVisual EvidenceTextual DebtPruningKullback-Leibler DivergenceSelf-distillationSupervised Fine-tuningContext ManagementTool-use Agents
Authors
Yuchen Huang, Sijia Li, Jun Zhang, Yi R. Fung
Abstract
Multimodal Large Language Models (MLLMs) are increasingly deployed as multi-step agents, where explicit reasoning supports task decomposition and tool coordination but also accumulates self-generated text. Over long trajectories, this text can dominate the context and suppress visual evidence, creating textual debt. We observe that reasoning becomes redundant once task-relevant visual evidence is grounded, while stale hypotheses can misguide later inference when grounding remains uncertain. Pruning must therefore remove redundant text without discarding visual evidence. We propose SPARE, a Kullback--Leibler (KL)-guided framework for pruning accumulated reasoning in multimodal tool-use agents. SPARE uses a compact task-state summary as privileged diagnostic context. For each candidate segment, it replays the same model under the original and summary-conditioned contexts. Reverse-KL divergence from on-policy self-distillation (OPSD) then tests whether the summary sufficiently covers the segment without disrupting future reasoning. We further fine-tune the summarizer with supervised fine-tuning (SFT), enabling more compact summaries, broader coverage, and more aggressive pruning. Across multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods while removing 37.89--64.58\% of reasoning tokens. This favorable accuracy--context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.