Banking workflow agents face challenges completing tasks accurately and fully
Bank-World: A Benchmark for Evaluating Agents on Banking Workflows
Artificial Intelligence
Summary
Handling banking tasks is more than just getting the right numbers—it also requires keeping records straight, getting approvals, and adapting to changes. The authors created Bank-World, a set of 50 simulated banking tasks with two fake banks to test if AI agents can handle these complexities. They found that while AI can make some correct choices, they often mess up when finishing tasks, like charging fees twice or paying before approval. Different AI models had different strengths in how well and how consistently they managed the tasks.
What this means in practice
- •For banking software developers: Assess AI agents’ ability to correctly complete complex banking tasks involving approvals and record reconciliation in simulated environments.
- •For enterprise automation teams: Evaluate and compare AI workflows for executing multi-step transaction tasks requiring interaction and state management across departments.
Authors
Daniel Rupawalla
Abstract
Bank-World is a benchmark of 50 banking tasks across two simulated institutions. Completing banking work requires more than producing the right calculation: agents must reconcile records, obtain permission, execute transactions, and keep instructions and status reports consistent as circumstances change. Bank-World tests these requirements through single-turn assignments and multi-turn interactions with simulated employees who supply information, approve actions, and revise instructions. Grading combines executable checks of recorded actions and state with model judgments of task requirements, including whether valid alternative solutions satisfy them. Across 1,600 attempts by four models, we find that correct decisions often fail to become completed work. Agents charge fees twice, distribute the correct total in incorrect amounts among the recipients, and execute payments while approval is pending. Their strengths also differ: Sol leads single-turn completion, Opus has the highest observed multi-turn completion rate, and Grok succeeds across more multi-turn tasks but less consistently. Bank-World makes these differences inspectable through task-level results and worked examples connecting source evidence, agent decisions, and simulated consequences. We retain the task environments, executable graders, and full trajectories privately to reduce contamination and preserve the benchmark for future evaluations.