How bert cross-encoders understand relevance between queries and documents
How BERT-based Cross-Encoders Model Relevance in Information Retrieval?
Information Retrieval
Summary
Matching queries with relevant documents is important but sometimes hard to understand for advanced AI models like BERT-based cross-encoders. This paper looks inside these models to see how they figure out if a document answers a question, including how they handle different types of queries and documents they weren’t trained on. The authors analyze multiple versions of these models to identify the key signals they use for matching. Their results help explain why these models perform well and how they might be improved or better applied in practice.
What this means in practice
- •For search engine engineers: Improve relevance scoring by understanding the internal signals cross-encoders use to match queries with documents in search systems.
- •For digital library developers: Design more robust retrieval models that maintain performance when faced with document or query types outside their original training.
Authors
Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski
Abstract
Cross-encoders are the most effective but also the least efficient models in Information Retrieval. Despite their success, the mechanisms underlying their relevance predictions remain poorly understood, particularly how they process to match queries and documents. This lack of interpretability limits their optimization, can lead to suboptimal retrieval, especially out-of-domain, and raises efficiency concerns for over-parameterized models. To address these challenges, this paper investigates two under-explored aspects of cross-encoder behavior. First, to understand the relevance prediction process, we trace the flow of information at the different stages of the process and between different input segments. We study the nature of the signals that compose this flow and that allow cross-encoders to predict the relevance of a passage to a query. Our work particularly focuses on the matching performed by the cross-encoders between queries and documents. Second, we explore how cross-encoders behave with out-of-domain documents and/or queries. Through a series of interpretability experiments, we evaluate three BERT-based cross-encoder models, varying in size and pre-training strategies, across both in-domain and out-of-domain datasets. Overall, this study both consolidates what is known about how cross-encoders model relevance and extends it by exploring the role of matching signals.