Ai benchmarks miss real work challenges in organizations
Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking
Computers and SocietyArtificial Intelligence
Summary
Many tests for AI focus only on solving technical problems but miss out on the harder parts of working inside organizations. The authors show that real work involves dealing with teamwork, rules, time limits, and changing situations. They created a way to describe these organizational challenges and found that most AI tests do not cover them well. They suggest future AI tests should look at how well AI handles these real-world work conditions.
What this means in practice
- •For enterprise ai teams: Design AI evaluation methods that measure how well AI systems handle complex organizational interactions beyond technical tasks.
- •For workflow software developers: Incorporate organizational factors into AI integrations to better support real-world work environments and task interdependencies.
A position paper. It proposes an approach and reports no results.
Authors
Victor Barres, Aden Barton, Nicolai J. Foss, Henrik M. Nielsen
Abstract
AI benchmarks have made remarkable progress in measuring the ability of AI systems to perform increasingly demanding cognitive and technical tasks. Yet, many of these benchmarks treat technical tasks as the unit of analysis, while much economically valuable work takes place within organizations, where tasks are embedded in a system of relations. Within organizations, successful task performance depends not only on solving the technical demands of the task but also on navigating the interdependence of activities, distributed knowledge, differentiated authority, temporal constraints, conflicting interests, and changing environmental conditions while the task is underway. Accordingly, we argue that organizational task difficulty constitutes a distinct and insufficiently recognized dimension of AI capability, and that existing approaches to benchmarking provide an incomplete basis for assessing how effectively AI agents can perform real-world organizational work. By drawing on organization theory, we develop a taxonomy of the structure of work that characterizes the conditions that agents face in organizational work. Applying this taxonomy to analyze existing benchmarks, we find limited and uneven coverage of organizational dimensions, with fewer than one quarter of prominent benchmarks claiming any of them. We therefore argue that AI benchmarks should increasingly move beyond the evaluation of technically difficult tasks toward the systematic assessment of agents' ability to navigate the organizational conditions under which economically valuable work takes place.