Jev improves Brazilian Portuguese search accuracy and ranking quality
Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking
Information Retrieval
Summary
Search engines need to decide which documents are most relevant to your query. This study looks at Jev, a fast decision-making model, to judge how well documents match a search in Brazilian Portuguese. The researchers found Jev's judgments mostly agree with human experts and that using Jev to reorder search results greatly improved the ranking quality. The model worked across different topics and made search results more useful.
What this means in practice
- •For search engine developers: Use Jev to automatically assess document relevance and improve search result rankings in Brazilian Portuguese collections.
- •For digital library managers: Apply Jev to enhance document search effectiveness across diverse Brazilian Portuguese domain-specific archives.
Authors
Leandro Carísio Fernandes, Jayr Pereira
Abstract
This study investigates Jev, a System One model designed for fast, structured decision-making, in information retrieval. We use Jev-as-a-judge to generate relevance judgments for four Brazilian Portuguese IR collections covering different domains and compare them with the official relevance judgments of each collection. We also evaluate Jev as a reranker. For relevance assessment, exact agreement ranged from 52.9% to 64.9%, while quadratic weighted Cohen's kappa ranged from 0.52 to 0.82. Most disagreements involved adjacent relevance levels, with at least 94.5% of annotations matching exactly or differing by only one relevance level. Despite differences at the individual query--document level, Jev-generated qrels produced similar retrieval system rankings, with Kendall's tau ranging from 0.90 to 0.93 for NormasTCU and Ulysses-RFCorpus across the two evaluation metrics. For reranking, Jev reordered the top 1,000 BM25 candidates for each of the 50 Quati test queries, increasing nDCG@10 from 0.3329 to 0.6331 (+90.2%) and MRR from 0.6852 to 0.9033 (+31.8%). These findings support Jev as a relevance assessor and show improved ranking effectiveness over BM25 in the evaluated reranking setting.