Tone changes how AI judges relevance in search results
Should I Be Polite to My LLM Relevance Judge? Tone as a Severity Operating-Point Shift
Information Retrieval
Summary
Some large language models judge how well search results match a query, but their decisions can change based on how the prompt is worded—especially the tone used. The authors studied different politeness levels and found that changes in tone mostly shift how strict or lenient the model is, rather than making it better or worse at judging relevance. This means being polite or not can make the model act more or less harsh, affecting agreement with human judgments. These findings explain past mixed results and warn that tone can affect the trustworthiness of relevance labels from AI judges.
What this means in practice
- •For search engine developers: Adjust prompt tone to control AI relevance judge strictness, improving alignment with human relevance assessments.
- •For chatbot builders: Design polite prompts to influence AI relevance judgments when ranking or filtering generated outputs.
Authors
Tian Zhang, Meng Li
Abstract
Large language models are increasingly used as relevance judges, yet their labels can shift with prompt surface form. We study one such feature -- tone -- on 3,498 TREC DL19/DL20 query-passage pairs, across eight judge models, five classifier-calibrated politeness levels, and three paraphrases per level. Effects are strongly model-dependent: one judge shows a structured U-shaped response, whereas most show only small changes. Where tone changes agreement, the results are more consistent with a shift in the judge's severity operating point -- its overall scoring leniency -- than with improved judgment. Agreement rises or falls as this shift moves the judge toward or away from human annotators' strictness. A query-disjoint cross-fit retains the expected association (Spearman $ρ= -0.683$; exact model-block permutation $p = 0.019$). Tone affects calibration-based agreement more than ranking outcomes: across 32 model-tone contrasts, the largest absolute mean change in NDCG@10 is 0.011, although Kendall's $τ$ as low as 0.743 shows that reordering is reduced, not absent. The account reconciles prior contradictory findings and identifies prompt tone as a potential validity threat when absolute relevance labels matter.