Occupation-focused benchmark tests large language models on real work tasks
ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge
Computation and LanguageArtificial Intelligence
Summary
It can be hard to test how well AI programs understand real jobs because good questions are expensive and rare. The authors created ORQA, which uses trusted job-related websites to make real work questions for AI to answer. They tested many popular AI models and found some do quite well on healthcare jobs but struggle on others like office support. This method helps see what kinds of professional knowledge these AIs really have and where they still need improvement.
What this means in practice
- •For enterprise ai teams: Assess large language model abilities on real-world job tasks across diverse occupations using an occupation-specific benchmark.
- •For hr tech developers: Design AI-based candidate screening tools that evaluate response accuracy to occupation-relevant questions from trusted professional sources.$Commercial implications: Enables creation of reliable hiring assessment products that use occupation-specific question-answering powered by language models.
Authors
Shreyas Krishnan, Serina Chang, Abhishek Nagaraj
Abstract
We present ORQA, a method for testing occupation-level knowledge in large language models. Prior methods either map abstract LLM skills to occupations via task definitions or utilize expert knowledge which is difficult to obtain at scale and expensive. ORQA complements both of these methods by connecting O*NET occupations to trusted occupation-specific websites (such as regulatory agencies, licensing bodies, professional organizations, and government publications) and converting these into source-traceable question-answer pairs. A combination of an automated pipeline and human review produces a set of high quality questions about occupations. The question set created via our method covers 116 occupations from all 21 major groups in the SOC, with 480 questions sourced from 187 different websites. Each question is designed to probe a real-world skill question that is relevant to the occupation in question. We test 15 state-of-the-art frontier and open-weight models via this method. Claude Opus 4.6, GPT-5.4 and Claude Sonnet 4.6 all perform the best at approximately 58-62% while smaller open-weight models achieve approximately 33-41% performance. Performance varies significantly across occupations. Healthcare-related occupations achieve the highest performance (78%) while Office and Administrative Support achieve approximately 40%. Performance on individual occupations (e.g. Sheet Metal Workers and Fish and Game Wardens) is essentially zero. We also find that open-ended questions and weighting by wage bill do not significantly affect the ranking of models on this benchmark. We believe that leveraging existing trusted occupation-specific information to test LLM knowledge in professional domains may be a scalable and useful method for evaluating occupation-level AI performance in the future. Results and data are available at orqabench.org.