Decision model controllers increase agent cost without success benefit

Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

Artificial IntelligenceMachine Learning

Summary

This paper looks at whether adding a special decision helper (called Jev) to an AI agent would make it cheaper or better. The authors tested Jev against other simpler methods to see if it improved performance or cost. They found that Jev made the agent cost more but didn’t improve how often it succeeded. Even though Jev gave better signals offline for when the agent was done, it wasn’t good at knowing when the work was actually complete. This means that having fancy decision models doesn’t always help make AI agents better or cheaper.

What this means in practice

  • For ai system engineers: Assess cost-effectiveness of adding decision-model controllers when designing multi-step AI agents to avoid unnecessary expense without success gains.
  • For automation platform developers: Use insights on controller calibration limitations to design better verification steps without raising costs in automated task-running systems.

Authors

Naman Vats

Abstract

Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse. In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments. On 21 held-out tasks (315 trials), Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at an identical pass rate (73.0% vs. 73.0%). Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct. No controller ever chose to abandon a run. Informative decision signals did not become useful interventions; we discuss what that would require.