WorkWorlds measures AI performance across real workplace contexts
WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks
Artificial Intelligence
Summary
Many tests for AI measure how well it completes specific workplace tasks, but these tests often give the AI exactly the information it needs upfront. This can make the AI's job easier than in real life, where information is scattered and must be found first. The authors built WorkWorlds, a system that separates the workplace environment from the tasks, so AI agents must find necessary information before completing tasks. Testing this in a simulated company showed that when the AI had to access all usual information first, its success rate dropped compared to when the environment was specially prepared with task information.
What this means in practice
- •For ai development teams: Use WorkWorlds to benchmark AI agents’ ability to find and use workplace information in realistic settings rather than curated tasks.
- •For business process analysts: Simulate organizational roles and tasks to identify how AI support tools perform across full workflows and information access scenarios.
Authors
Yining Hua, Levi Lian
Abstract
Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task has been specified. This design measures performance on workplace-like tasks in an environment assembled for the task. When task specification guides which context is selected, the evaluation can encode task information into the environment and pre-complete part of the information-localization work that workplace performance normally requires. We introduce WorkWorlds, an evaluation infrastructure that separates organizational state from task specification. A world first fixes a revision, date, and employee seat and materializes the organizational state that employee can access; tasks are introduced only afterward. We implement WorkWorlds in a primary synthetic pharmaceutical company with 8 measured tasks across 6 employee seats, and construct additional organizational worlds. Across 192 matched evaluations, moving from task-curated context to the full role-visible workplace reduced evidence access from 90.4% to 74.5% and criterion pass from 79.4% to 68.2%, while pass conditional on evidence access remained nearly unchanged; most of the measured difference occurred before the agent reached sufficient evidence.