Learned context planning falls short of strong retrieval in long QA
When Learned Context Planning Fails to Beat Strong Retrieval: A Controlled Study of Planning, Routing, and Reranking for Long-Context QA
Computation and LanguageInformation Retrieval
Summary
Answering questions using very long documents is hard because you have to pick the right bits of information first. This paper tested if teaching a model to select helpful pieces of context before answering helps more than just using strong search methods. The authors found that learned planning methods do not outperform strong retrieval techniques for long multiple-choice question answering. Their tests suggest it’s better to rely on good retrieval than on learned planning in this setting.
What this means in practice
- •For software engineers: Design QA systems that prioritize strong retrieval methods over learned context planning for better accuracy with long documents.
- •For information retrieval teams: Optimize evidence selection pipelines in long document QA by focusing on retrieval and reranking techniques rather than learned planning.
Authors
Yingrui Li, Han Chen
Abstract
Learned context planning selects evidence atoms before an answer model reasons over them. We test whether this learned selection improves long-context multiple-choice QA after strong retrieval, routing, budgeted-selector, and reranking controls. Our primary diagnostic uses all 503 LongBench-v2 MCQ questions with Qwen2.5-7B-Instruct. The planner is SFT-trained on outcome-selected traces from 140 training and 28 development questions; because the 503-question analysis includes those questions, it is partly transductive. At an 18k-character budget, anchored hybrid retrieval reaches 36.18% accuracy and BM25 reaches 35.98%, while the best direct planner-guided method reaches 34.19%. On the untouched 152-question test split, anchored hybrid remains higher (42.11% versus 36.84%). Leakage-safe routers cannot convert a large oracle gap. Under tight budgets, the best planner is ahead by only 0.40 points at 6k and loses at 9k; planner-guided reranking has a +1.79-point estimate at 6k with a paired interval crossing zero and ties the control at 9k. Packing-order and score-flatness analyses did not identify a stable mechanism. Under this setup, learned planning is a weak relevance signal rather than a replacement for strong retrieval.