Selective token budgeting improves financial document question answering
Token-Budgeted Escalation for Financial Document QA: Cost Is Predictable, Benefit Is the Bottleneck
Information Retrieval
Summary
Answering questions from financial documents can be costly because each question might need different amounts of information to get a good answer. The authors studied how to use a limited amount of tokens (a measure of information processed) smartly by deciding which questions deserve more detailed searches. They found that predicting the cost of extra detailed searches is accurate, but knowing which questions will benefit most from going deeper is still challenging. Using their method, they improved answer quality while saving many tokens compared to always doing deep searches.
What this means in practice
- •For financial data teams: Manage token usage when answering finance document queries to improve response quality without exceeding processing limits.
- •For legal document analysis teams: Allocate computational resources effectively for complex document question answering where query costs vary widely.
Tested on one dataset.
Authors
Junru Zhu, Yixin Yang, Xiaoqing Ding, Ruoyu Qi
Abstract
Retrieval-augmented generation systems can route difficult queries to deeper context, but batch deployments must allocate a shared token budget across calls whose costs vary by query. We formulate selective escalation as finite-batch allocation for financial document question answering. Each of 150 FinanceBench questions first receives a top-1 retrieval answer. Predictors estimate the adjudication-quality gain and token cost of an optional top-5 call, and the allocator prioritizes calls by predicted gain per token. At the nominal 10% budget, gain-per-token allocation improves adjudication quality over gain-only ranking by 0.034 (95% document-bootstrap CI [0.001, 0.072]) while using 46.6% fewer total tokens than one-pass top-5 retrieval. Additional-call cost is accurately predictable (R-squared 0.93), whereas beneficial escalation remains difficult to rank (AUROC 0.60). These results show that heterogeneous cost is actionable under tight constraints, while progress across the full budget frontier depends on stronger query-specific benefit estimates.