MORSE method improves evidence retention by reordering contexts
MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression
Computation and Language
Summary
Sometimes when computers try to compress information from multiple sources, the order in which the context is arranged changes how well important evidence is kept. The authors found that putting stronger, more relevant information first helps prevent losing key details. They created MORSE, a method that reorders information intelligently before compression to keep important evidence safe. Tests show MORSE keeps more crucial facts and improves performance in question-answering tasks.
What this means in practice
- •For machine learning engineers: Improve multi-source data compression to retain critical evidence for better AI question answering performance.
- •For document summarization teams: Increase the accuracy of compressed documents by reordering context to preserve key supporting information.
Authors
Ke Wan, Yifan Wang, Liheng Lai, Chen Chen
Abstract
Likelihood-based context compression can account for cross-context redundancy through sequential scoring, but this makes compression outcomes sensitive to context order. We show that different permutations of the same context collection can produce markedly different evidence-retention outcomes under an unchanged compressor. We attribute this sensitivity to information preemption: earlier partially relevant contexts can absorb credit for shared information, suppressing the incremental score of later, stronger evidence carriers and increasing their risk of removal. Controlled pair-swap interventions directly support this mechanism by showing that evidence-first ordering substantially improves supporting-evidence survival. To address this problem, we introduce MORSE, a compression-aware method for evidence-preserving context ordering. MORSE applies a common reverse query-evidence principle to both individual contexts and compressed candidate outputs, using the former to construct an evidence-first anchor and the latter to guide compression-aware permutation selection. Across multi-hop QA benchmarks, compression procedures, budgets, and scoring models, MORSE consistently improves evidence preservation over static reverse ordering and compute-matched random search, with corresponding overall improvements in downstream QA. Our code is available at https://github.com/tbn5pj/MORSE_code.