Cross-Attention Calibrated Deduplication for Retrieval-Augmented Generation System
2026-07-27 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors found that existing methods for removing duplicate text chunks in Retrieval-Augmented Generation systems either don’t catch all duplicates or are slow. They introduced a new way called Cross-Attention Calibrated Deduplication (CACD), which compares chunks with more detailed, token-level information to better spot duplicates. Their method also uses a score to see how much new information a chunk adds and takes a vote across several comparisons. In tests on a standard dataset, CACD removed a similar amount of duplicates as other semantic methods but was faster than them. The authors share their code for others to try and improve upon.
Retrieval-Augmented Generationchunkingdeduplicationvector databasecosine similaritycross-encoderattention entropyNew Information Scoresemantic filteringSQuAD dataset
Authors
Phuong Le Huy, Nam H. Nguyen, Quan V. Dang
Abstract
Common chunking strategies in Retrieval-Augmented Generation (RAG) systems often create redundant chunks. These redundant chunks make the vector database bigger and slow down retrieval. A common fix is cosine-similarity thresholding. This method reduces each chunk to a single vector, then compares vectors using a similarity score. But a single vector can lose the fine-grained, token-level detail needed to tell a true duplicate apart from a chunk that just shares the same topic. We propose Cross-Attention Calibrated Deduplication (CACD). CACD checks each new chunk against an in-memory pool of chunks already kept, using a cross-encoder instead of a single pooled vector. This keeps token-level detail all the way to the final comparison. CACD combines three parts: the cross-encoder comparison itself, a New Information Score (NIS) that measures how much of a chunk is not explained by a candidate already kept, and a majority vote across several candidates rather than a single best match. NIS is calculated from the attention entropy of the cross-encoder. We tested CACD against five existing filtering methods, nine chunking strategies, and 18 configurations, all on the full SQuAD 1.1 validation set. In our experiments, CACD removes 9.75% of chunks on average. This drop rate is close to other semantic-level methods, and much higher than exact-match filters, which barely remove anything. In these experiments, CACD also processes each configuration in 51.0 seconds on average, about 27% faster than the strongest baseline, NERExact (69.6s), and about 7x faster than cosine-similarity filtering (356.7s). These results come from a single dataset, so we present them as an early comparison, not a general claim. Code for the baseline evaluation and for CACD is available at https://github.com/lehuyphuong/rag_bench and https://github.com/lehuyphuong/cacd_dedup.