Autonomous agents’ reports reveal partial story of their actions
Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agents
Human-Computer Interaction
Summary
When coding agents finish tasks, developers read summaries the agents write about themselves instead of detailed logs. The authors studied thousands of real sessions and found that these summaries include only a small portion of the agent’s actions and only partially reflect what actually happened. Interestingly, when agents stray more from their original plan, their summaries tend to refer more to that plan. The study carefully verified the measurements using language models to make sure only reliable results were reported.
What this means in practice
- •For software developers: Improve debugging by focusing on the limits of agent-generated summaries versus full action logs during task reviews.
- •For automation engineers: Design better monitoring tools that account for disparities between agent reports and actual actions, especially when agents deviate from their plans.
Authors
Obada Kraishan, Kulsawasd Jitkajornwanich
Abstract
When a coding agent finishes a task, the developer reviews a summary the agent wrote about itself, not a display someone designed. We ask how much of the agent's work that summary carries, and whether it drifts toward the plan the agent stated when execution departed from it. Across 5,851 real developer sessions and 355,942 tool calls, a self-report referred to about one action in eleven, and a reader working from the report alone recovered roughly a fifth of the action log. Neither figure depended on whether the session later needed human correction. Reports did not generally resemble the stated plan more than the executed one, but they did so increasingly as execution diverged from the plan. We hand-validate both measurement steps that use a language model, report the one that failed alongside the one that passed, and draw conclusions only from measures that survived.