CascadeFilter reduces cost and boosts accuracy in semantic data filtering
CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection
Databases
Summary
Semantic filtering helps find documents that match a natural language request but running a big language model on all documents is expensive. The authors propose CascadeFilter, a three-step method that uses smaller models first and only forwards uncertain cases to a bigger, more costly model. Their method carefully picks thresholds for each step to minimize cost while keeping accuracy close to the oracle, the most trusted model. This approach improves evidence extraction and routing and shows better accuracy on many datasets compared to prior work.
What this means in practice
- •For data analytics teams: Reduce computational costs while maintaining accuracy when filtering large document collections with natural language queries.
- •For enterprise search engineers: Implement cascaded semantic filters combining small and full language models to efficiently process large-scale search and retrieval tasks.
Authors
Yanji He, Tao Zhong, Hongyang Chen, Xiaobo Zhang, Mingyu Yang, Wei Wang
Abstract
Semantic filtering selects records that satisfy a natural-language predicate and is an important operation in data analytics based on large language models (LLMs). Executing an expensive model on every document incurs substantial inference cost. Existing approaches reduce this cost by accepting predictions from cheaper models and forwarding uncertain records to an oracle. Task Cascades further varies document portions and operations, and provides an accuracy-guaranteed extension. However, its threshold calibration follows a prescribed adjustment sequence after cascade assembly, rather than minimizing cost over alternative combinations of stage thresholds. Motivated by this, we propose CascadeFilter, a three-stage semantic filtering method that jointly searches the thresholds for a small language model operating on extracted evidence and full documents, with unresolved records forwarded to the oracle. For a fixed cascade, we formulate threshold selection as minimizing token-aware cost subject to an oracle-relative error constraint, and select the least-cost certified configuration in a pre-specified grid. A finite-population calibration procedure bounds oracle-relative error after selection, using fixed-sequence tests on threshold tracks to allocate the multiple-testing budget. We further combine lexical and dense retrieval, reranking, and neighborhood-supported confidence to improve evidence extraction and routing. These components enhance the cascade while leaving its certification interface unchanged. Across six datasets and 13 evaluation settings, the reported results have the highest benchmark accuracy and macro-averaged F1 score in 12 settings each. These comparisons describe the reported operating points; the accuracy guarantee concerns agreement with the oracle.