Tie-breaking rules change outcomes in item recommendation rankings
Tie Handling Is Part of the Evaluation Protocol: An Order-Invariance Audit for Tie-Heavy Recommender Scores
Information Retrieval
Summary
When computer programs suggest top items to users, several items can have the exact same score, leading to ties. The way these ties are broken can change which items appear at the top, even if the scores do not change. The authors found that different tie-breaking rules drastically altered evaluation scores in common recommendation tests. They also provide a way to measure these effects and a checklist to report tie-breaking methods. This issue matters in many recommendation tests where tied scores influence which items are shown.
What this means in practice
- •For machine learning engineers: Assess recommendation model rankings reliably by accounting for tie-breaking methods that affect item order in evaluation.
- •For data scientists: Improve comparison of recommendation algorithms by reporting and auditing tie-handling procedures in offline evaluation metrics.
Authors
Chengkun Guo, Han Chen, Yilin Zhu, Yingrui Li
Abstract
Offline top-k evaluation often ranks one held-out relevant item together with sampled negatives. When several candidates receive exactly the same score, the tie-breaking rule becomes part of the ranking. A common implementation stores the relevant item first and then applies a stable sort, which preserves input order among equal scores; the relevant item therefore wins every tie. We call an evaluator row-order invariant when permuting the input candidates without changing their identities, labels, or scores leaves the final ranking unchanged. We audit this property by holding candidates and scores fixed and changing only the tie-breaking rule. On 30,000 Amazon Beauty & Personal Care rows, NDCG@10 for a rating-weighted attribute-overlap score is 0.85 under input-order tie-breaking. A deterministic hash tie-break based on user and item IDs lowers it to 0.17. The exact expectation under uniform random tie-breaking closely matches the mean over 100 independent hash seeds, while a residualized attribute score with few exact ties is nearly unchanged. MovieLens Tag Genome shows the same pattern for an attribute-overlap score, whereas item popularity is nearly unchanged. We derive expected Hit Rate and NDCG at cutoff k when the relevant item is randomly ordered among candidates with the same score, and we provide a practical reporting checklist. The same issue can occur in sampled or full-catalog evaluation whenever exact ties affect top-k membership or rank.