Skill extraction models graded to improve job recommendation quality
Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026
Artificial Intelligence
Summary
People often score skill-finding computer models as right or wrong with just a few examples, missing how well the model really understands skills. The authors show a new way to score skill predictions with multiple grades, capturing how close or relevant each predicted skill is. They use a series of automated yes/no questions answered by a smaller AI to decide these grades, which they checked against humans and top AI models. Their tests found that better models are good at avoiding nonsense skills rather than just finding more right ones. The paper shows there is still plenty of work to make skill recommendation smarter and more efficient.
What this means in practice
- •For recruitment platform engineers: Improve job matching quality by integrating graded skill relevance scoring into large-scale skill extraction models.
- •For human resources software teams: Develop better skill recommendation systems that reduce irrelevant skills in candidate rankings for job postings.
Authors
Warre Veys, Luis Gasco, Jens-Joris Decorte
Abstract
Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the annotation methodology, the construction of five benchmark datasets (four with vacancy sentences as queries, one with lexically distant skill surface forms), the validation of the LLM cascade, and insights from the submissions. The field converged on multi-stage retrieve-and-rerank pipelines, whose gains come from clearing nonsense out of the ranking window rather than finding more positives: at the top of the leaderboard, the graded ordering is uncorrelated with what binary evaluation would have said. Graded labels expose much bigger gaps between top submissions and baselines than binary evaluation does, and show that semantic-understanding skill recommendation is still wide open. The clear next direction is bringing efficiency back: every submitted system we could size uses at least 42x, typically several hundred times, more parameters than the baselines it beats.