Context aware search shows improved accuracy over keyword methods

Towards Semi-Automatically Comparing Keyword-Based and Semantic Search Accuracy

Information Retrieval

Summary

Finding relevant information in large collections can be tricky. Traditional searches use keywords but sometimes miss context, whereas newer chat-based methods try to understand meaning better. The authors created a way to compare both search types fairly and quantitatively, even though they give results in different styles. They tested this method in a real industrial setting and found that context-aware chat searches retrieved more complete and accurate information than keyword searches.

What this means in practice

  • For enterprise search teams: Evaluate and improve company-specific search tools by quantitatively comparing traditional and chat-based methods within their domain.
  • For software developers: Integrate semi-automatic metrics from the framework to benchmark semantic search features against keyword matching in apps serving complex queries.

Authors

Mohamed Ben Salha, Fiete Lüer, Maik Betka, Stefan Wagner

Abstract

The increasing importance of Information Retrieval (IR) in managing large datasets has highlighted significant limitations in traditional keyword-based search systems. Context-aware chat-based search methods, such as Retrieval Augmented Generation (RAG), have recently emerged, but their evaluation compared to keyword-based systems often relies on subjective user feedback. A rigorous, quantitative comparison between these paradigms remains lacking. This work introduces a novel, preliminary framework to quantitatively assess IR accuracy of search systems that produce different output formats, such as lists and messages. It focuses on two key aspects: the ranking accuracy for keyword-based systems and the completeness of retrieved information for semantic chat-based systems. Our approach enables semi-automatic comparisons of semantic and keyword-based methods using interchangeable equivalence classes tailored to domain-specific contexts (e.g., companies or problems). We validate the framework through an industrial case study, demonstrating statistically significant improvements in context-aware search over keyword-based methods, supported by analyses including the Mann-Whitney U-Test. With its adaptable design, the proposed framework provides a strong foundation for objectively assessing keyword-based and semantic chat-based search methods.