Benchmark compares ways to keep important facts with limited text
CoRA-Bench: Evaluating Evidence Preservation by Context Selectors Under Token Budgets
Artificial IntelligenceComputation and Language
Summary
Long text inputs can be too big for computers to handle, so parts are cut out to fit token limits. The authors present CoRA-Bench, a test to see how well different methods keep the important evidence needed to answer questions when reducing text. They find that different methods work better for different types of tasks and evidence. For example, one method works best for financial text, while another is better for diagnostic data. Their results show that preserving key facts doesn’t always match up with how well answers can be given afterward.
What this means in practice
- •For natural language processing teams: Optimize selection of text snippets to keep key facts when processing very long documents under token limits.
- •For financial data analysts: Improve extractive question answering on lengthy financial reports by choosing context selection methods that retain critical evidence.
Authors
Huzaifa Pardawala, Siddhant Sukhani, Sudheer Chava, Agam Shah
Abstract
Context selection can reduce input length during long-input inference but can remove evidence required to answer a query. We introduce CoRA-Bench, a benchmark for evaluating evidence preservation under shared token budgets. We compare head and tail truncation, BM25, dense retrieval, dense retrieval with reranking, LLMLingua-2, and Provence across BABILong, LOONG Financial Spotlight, a custom NoLiMa diagnostic, and LongMemEval. Evidence-preservation results reveal task-dependent tradeoffs: BM25 achieves 0.921 supporting-fact recall on BABILong at 128K, dense retrieval and dense reranking each achieve 0.908 needle recall on NoLiMa at 64K, and dense retrieval achieves 0.993 answer-bearing-turn recall on LongMemEval at 64K. Evidence, token retention, and complementary reader evaluations use the same selected contexts. Reader evaluations use dataset-native answer metrics and three runs on the same questions, with paired bootstrap confidence intervals. At a 32K budget, BM25 achieves 46.20% BABILong answer accuracy and dense retrieval achieves 71.13% LongMemEval accuracy. The results show that method rankings vary across evidence types and between evidence recall and downstream accuracy.