Benchmark measures ai performance on long professional tasks
DAYJOB: A Benchmark for Long-Horizon Professional Work
Artificial IntelligenceComputation and Language
Summary
Professional jobs often begin with vague requests, requiring workers to figure out what to do and what information matters. The authors created DAYJOB, a set of 130 complex tasks from healthcare and finance that mimic real professional work lasting many hours. They tested many AI models on these tasks and found the best models only succeeded about a quarter of the time, showing these tasks are very challenging for AI. They also saw that AI sometimes trusts wrong information, leading to mistakes. The authors share the tasks and tools to help improve AI in handling long, realistic professional work.
What this means in practice
- •For enterprise ai developers: Evaluate and improve AI agents on realistic, long-duration tasks in healthcare and finance using a standard benchmark and strict scoring criteria.
- •For financial analytics teams: Use DAYJOB finance tasks to test AI systems for handling complex, multi-step financial work requests accurately before deployment.
Authors
Stephanie Finley, Liudas Panavas, Thomas Mikkelson, Cam Hinton, Stacey Ganss, Bradley Monton, Emily Kendall, Michelle Spradlin, Lydia Bye, Michael O'Brien, Lauren Ylvisaker, Derek Ray, Suhaas Garre, Sushant Mehta, Edwin Chen
Abstract
Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the request's premise holds. We introduce DAYJOB, a benchmark of 130 tasks built by professionals in healthcare (50) and finance (80). The tasks are estimated to take a professional 13.6 hours on average in healthcare and 16.6 in finance. Each task is a containerized Harbor environment with an expert rubric of binary criteria (median 47.5 and 57.5 per task) that an agentic judge applies to the delivered files, and an attempt passes only if it meets every criterion. Across 30 model configurations from 13 developers, the strongest, Claude Opus 5.5, passes 24.7% of healthcare and 23.9% of finance attempts, and the median configuration passes 0.6% and 2.5%. In case studies, agents accept premises that the record contradicts and carry wrong inputs through otherwise consistent analyses. We release all healthcare tasks, 50 of the 80 finance tasks, the evaluation harness, and the leaderboard.