Component replacement impact on local and task decisions in language model agents
What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents
Artificial Intelligence
Summary
This paper looks at how changing one part of a language model agent affects how it does tasks later on. The authors reviewed many studies and found that simply showing improvements in small decisions or in task outcomes does not prove that better small decisions cause better overall results. They explain that understanding these effects needs clear methods and controls. They also provide guidelines for reporting claims about these replacements.
What this means in practice
- •For ai system builders: Design clearer experiments when replacing components in language model agents to demonstrate which changes actually improve task outcomes.
- •For software testing teams: Use the review’s guidelines to better report and interpret testing results when modifying agent decision-making components.
A survey. It maps existing work.
Authors
Shuyang Zhang, Jianshuo Chang
Abstract
Background. A component replacement in a language-model agent changes an execution trajectory, potentially altering later observations, resource use, and recovery opportunities. Different evidence is needed to assess its task-level benefit and the contribution of local decision quality. Methods. This critical scoping review maps 348 studies and examines 90 comparison records: 88 from 40 included studies and two from supplementary studies. Eight purposively selected cases structure the synthesis around the replaced decision, executed conditions, measurement comparability, controls, and remaining explanations. Results. Of 222 studies reporting local decision metrics, 142 also report measured task endpoints and 49 report proxies. These counts identify studies that report both types of measurement, without establishing that the measurements come from matched comparisons. Outcome Monitors reports a package-level completion gain whose attribution to detector quality remains limited; First-chunk selection reports a local improvement assessed against an offline proxy endpoint; Evidence-Carrying Termination reports fewer premature unsupported terminations and completion non-inferiority, without establishing completion superiority. Cross-case analysis identifies three candidate mechanisms involving recovery and disruption, intervention timing, and downstream use. Attribution and deployment depend on the comparison controls, label definitions, and information available to the controller. Conclusions. The review distinguishes the task-level benefit of a component replacement from the contribution of local decision quality and derives eight claim-specific reporting items. Neither online execution nor simultaneous gains in local and task metrics alone establish that better local decisions explain the task-level gain.