Large language models struggle to reliably report task progress during work
The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?
Software EngineeringArtificial IntelligenceComputation and Language
Summary
This paper looks at how well large language models (LLMs) can track and report their progress as they work through tasks. The authors found that these models are good at reporting their status at the start and end of a task but often become less accurate in the middle. Different models fail in different ways during the task, and newer models improve some issues but become cautious near the finish. The study shows that relying solely on a model's self-reported progress to make decisions about continuing or stopping a task is risky. The authors offer a way to test models for this ability throughout a task's steps.
large language modelstask progress reportingagent frameworksprogress checkpointsmodel reliabilitytask execution stagesbenchmark evaluationstate reportsmid-task accuracytask flow control
Authors
Boyang Wang, Yunhan Wang, Yalun Wu
Abstract
Recent large language models can emit task-progress signals that agent frameworks use to decide whether a task should continue or stop, yet whether a model can reliably report its task progress at every stage of a task, and where and how its reports fail, has not been studied systematically. We evaluate this ability on the public benchmark $τ^2$-bench and on StageIF, a controlled testbed in which reporting checkpoints are placed across the task's lifecycle. Both settings require reports at multiple task stages. We find that reporting reliability depends on the stage a task has reached, and that almost every deployed model we test is reliable at some stages and unreliable at others. Where reporting breaks down is not the same everywhere. Most deployed models lose accuracy once work is under way and recover once the task is done. The newest generation closes that mid-task drop and instead grows conservative at the finish line. Our study exposes a capability gap in task-progress reporting and provides an evaluation protocol that spans the whole course of task execution for this ability on which agent operation depends. The findings indicate that agent frameworks should not control task flow on the strength of the model's state reports alone.