EviCast predicts value of evidence retrieval to improve language model tasks
EviCast: Forecasting Retrieval Gains Through Shared Response Calibration
Computation and LanguageMachine Learning
Summary
Sometimes language models use extra information to answer questions better, but not all extra information actually helps and can slow things down. The authors made EviCast, a method that predicts how useful extra evidence will be before trying it out fully, saving time and effort. EviCast combines how a model reacts to evidence with data about what different retrieval methods can provide. This lets people choose the best way to get information for a task without testing everything.
What this means in practice
- •For natural language processing teams: Assess retrieval strategies for question-answering systems to save computation by predicting which evidence improves results.
- •For data engineering teams: Optimize configuration of evidence retrieval modules within large-scale language model deployments by forecasting net utility before full execution.
- •For customer support platforms: Improve automated responses by selecting efficient evidence retrieval setups that truly enhance answer quality without unnecessary costs.$Commercial implications: Enables development of cost-effective, high-quality automated support agents by predicting best retrieval configurations.
Authors
Yibo Wang, Jisheng Dang, Bimei Wang, Jiongpei Hou, Jingwen Zhao, Wenjun Ma, Fei Shen, Juncheng Li, Meng Wang, Qi Tian, Bin Hu, Tat-Seng Chua
Abstract
Retrieval-augmented generation (RAG) extends large language models with external evidence, yet relevant evidence does not ensure that a generator can use it. Indiscriminate retrieval can degrade task performance while adding retrieval and long-context inference costs. Existing methods neither determine before large-scale execution whether a generator, task distribution, and retriever configuration is worth deploying nor diagnose multiple candidate retrievers under a shared budget. We introduce EviCast, a framework that forecasts retrieval value through interventional evidence-state decomposition. EviCast measures a generator's response to controlled evidence states once and combines this shared profile with the offline evidence-state distribution of each candidate retriever, yielding the end-to-end net utility and its 95% lower confidence bound without executing every RAG configuration. A state-refinement analysis characterizes when its binary and topology-aware instantiations are exact and when shared calibration is statistically more efficient than per-retriever calibration. In frozen evaluations, EviCast correctly decides all 14 HotpotQA retrieval configurations with a forecast error of 2.75 F1 points while reducing diagnostic generation calls by 80%, and it attains the lowest forecast error among seven predictors in the low-budget regime. Its topology-aware instantiation, EviCast-T, captures the complementarity between terminal and bridge evidence, correctly deciding all 16 configurations on MuSiQue and HotpotQA and 23 of 24 after prospective transfer to a third dataset. These results show that retrieval deployment depends on both evidence availability and the generator's ability to convert evidence structure into stable net utility. Code is available at https://github.com/YiboWang2002/EviCast.