Screen-based agents struggle with reliable enterprise software tasks
ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software
Artificial IntelligenceComputer Vision and Pattern RecognitionMultiagent Systems
Summary
Using computer agents to do tasks on business software, like managing inventory or billing, is hard because these systems have complex screens and multi-step operations. The authors created ERPBench, a test to see how well agents work on real enterprise software by checking if their actions actually save the correct information. They found that even if an agent seems to complete a task, it often saves wrong data. Their work shows that tools good at general computer tasks may not work well in business settings.
What this means in practice
- •For enterprise software teams: Evaluate and improve automated agents handling complex business software tasks by testing them against ERPBench’s real ERP data correctness metrics.
- •For software quality assurance teams: Use ERPBench’s human-approved execution harness to safely validate automation agents before deployment to reduce errors in enterprise workflows.
Authors
Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan, Yuying Zhao, Eugene Siow
Abstract
Computer-use agents that operate through screenshots and simulated actions are advancing rapidly, yet their evaluation remains anchored to general desktop and web tasks. Enterprise Resource Planning (ERP) systems run the finance, procurement, inventory, and customer operations of organizations worldwide, and pose distinct challenges for computer-use agents: dense interfaces, coordinated multi-step interactions, and errors that alter persistent business records rather than surfacing on screen. Existing enterprise benchmarks rely on proprietary platforms or on simulated approximations of such software. We introduce ERPBench, a benchmark that evaluates screenshot-only agents on a live and reproducible ERP system and scores each task against ground-truth values in its database. Beyond the benchmark, we present a production-grade harness that gates agent actions behind human approval for safe deployment, which ERPBench runs autonomously. Evaluating six closed and open-source agents, we demonstrate that strong general GUI performance does not transfer to enterprise reliability. Even when an agent reaches the right form and saves it, the stored record is often wrong: some agents save in up to 85% of runs but write the correct value in as few as 3%. We further characterize failure modes specific to enterprise workflows.