Humor ranking wins contest using multi-stage AI retrieval system
IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English
Information RetrievalComputation and Language
Summary
Judging how funny something is can be very tricky, so this team created a system called IROH to rank humor better. They combined different AI methods to pick the funniest sentences, including specialized judges that explain their choices. Their experiments showed that these explaining judges made the biggest difference, while some other techniques didn't help as much. Their smaller, well-tuned AI models sometimes worked better than bigger ones, which is surprising.
What this means in practice
- •For content recommendation teams: Rank humorous content more accurately in apps like social media or entertainment platforms using multi-stage AI judges with rationale explanations.
- •For chatbot developers: Improve humor understanding in conversational agents by integrating rationale-based large language model judges for better joke ranking and response selection.
Authors
Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truică, Elena-Simona Apostol
Abstract
Our team, VANGUARD, presents IROH (Insightful Ranking of Humor), a three-stage retrieval system for JOKER Task 1 English at CLEF 2026, achieving first place on the leaderboard with 0.6347 MAP. Our pipeline combines hybrid sparse-dense retrieval, cross-encoder reranking, and a LoRA-adapted Large Language Model judge ensemble. We employ Gemma 4 to generate query-aware rationales under two prompt strategies, generic and typed, and produce up to four types of structured hard negatives for training data construction. Through an ablation across three cross-encoder architectures, four dense embedders, and eight judge configurations, our key findings are threefold: (1) the rationale-distilled judge is the primary driver of ranking quality, whereas appending rationales to the first-stage index contributes negligibly; (2) structured hard negatives degrade generalisation in nearly all configurations despite inflating local validation scores; and (3) across the components we ablate, the lighter, better-calibrated model is competitive with or stronger than its larger counterpart, with the generic-rationale Qwen2.5-7B judge (0.6055 MAP) outperforming every Gemma-4-31B configuration, and the advantage of generic over typed rationales is concentrated almost entirely in the smaller model.