ShowTellArena benchmark tests business workflow understanding from demonstrations
ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations
Artificial Intelligence
Summary
People often teach others by showing how to do a task and explaining each step. The authors created ShowTellArena, a set of recorded business tasks with spoken explanations and questions to check if a computer system really understood the lesson. The tasks cover things like finance and hiring, and the questions test if the system learned the rules and exceptions. Their initial tests showed some mistakes and missed parts in how systems learned from the lessons. The dataset and test process are open for others to use and improve.
What this means in practice
- •For business process automation teams: Assess how well automation systems learn operational workflows from narrated demonstrations to improve task accuracy and completeness.
- •For workflow software developers: Use the benchmark and dataset to test and refine interfaces that teach business tasks effectively through demonstration and explanation.
Authors
David Garg, Ritobrata Sarkar, Ehsan Azarnasab, Siddhartha Borah
Abstract
We often teach a colleague by showing the work and explaining the decisions as we go. How can we check what an agent understood from the same lesson? We introduce ShowTellArena, a benchmark protocol and public dataset for comprehension after narrated business demonstrations. The v1.0 release contains 50 business workflow tasks, with recordings, screenshots, narration, fixture seeds, and 502 questions. Tasks span finance, hiring, procurement, customer decisions, inventory, and logistics. The protocol holds the business scenario and quiz fixed while allowing each product to capture the lesson through its own teaching interface. Questions test operational rules, boundaries, exceptions, and errors in proposed automations. We analyze 218 selected pilot attempts across 39 workflow cases, including 28 cases attempted by all three evaluated systems. These exploratory results expose both answer errors and failures to complete the teaching experience. We describe the release's verification gaps and the pilot's uneven coverage, exclusions, and grading provenance. The contribution is an inspectable dataset and assessment workflow that others can extend; the selected pilot is not a controlled product ranking.