Improving process predictability and accuracy with stepwise AI execution
One Step at a Time: Trading LLM Autonomy for Process Predictability
Computation and LanguageArtificial Intelligence
Summary
Many organizations use AI agents to automate tasks, but often lose track of how each step happens because the AI only shows the final answer. The authors designed a system that feeds the AI one step at a time, making the whole process predictable and easier to check. This method helps especially smaller AI models perform better and stops them from guessing answers without following the right steps. It also creates detailed logs for auditing and improving the process later.
What this means in practice
- •For business process automation teams: Deploy stepwise delivery of instructions to AI agents to ensure predictable and auditable task automation workflows.
- •For quality assurance engineers: Inspect detailed step output logs generated during AI-driven process execution to audit and optimize workflows.
Authors
Hans Schabert, Christoph Peters
Abstract
Organizations automating operational processes need more than a correct outcome: they need to predict how a process will run, know which one actually ran, and inspect it step by step. When an agent is the executor that predictability is normally lost: the prescribed procedure goes into the system prompt, and only a final answer comes back. We deliver the procedure step by step over the Model Context Protocol (MCP) instead: a server releases one step at a time, the agent executes it, and each step returns a structured step_output. This trades autonomy for predictability, and two properties then follow by construction, independent of the executor. The execution path is prescribed before the run, so the process is predictable in advance rather than reconstructed afterwards; and the completed step records form a machine-readable execution log that downstream tooling can audit and optimize step by step. Evaluating 15,475 trials across 13 SOP-Bench domains and four open-weight executors from frontier (Kimi K2.5) to lightweight (Ministral 3 8B), we find step-level delivery makes the executed process predictable and inspectable for every executor, and additionally raises accuracy when the executor is small. Across all four, process adherence rises significantly (76-95% to 95-99%) and ungrounded answers (correct outputs produced without executing the SOP) near-vanish, falling from 2.1-4.5% to 0.2-0.3% of trials (all 95% CIs exclude zero); under prompt-based delivery, 31-49% of correct answers on know_your_business bypass the SOP entirely, even for the frontier executor. Accuracy is where the executor's capability enters: the lightweight executor gains +6.5pp grounded accuracy because supplying the process externally removes a reconstruction burden it cannot carry, while capable ones trade a small raw-accuracy decrement for a predictable, auditable process.