TACIT-Switch: Cost-Aware Model Escalation for LLM Agents from Censored Supervision
Machine Learning
Summary
The authors study how to balance using smaller, cheaper language models that sometimes fail with larger, more reliable but expensive ones. They propose a method called TACIT-SWITCH, which learns when to switch from a small to a large model based on past examples of where smaller models tend to fail. This method doesn’t need a teacher during actual use and improves success rates by about 7-11% compared to other switching strategies in simulations. It performs well on tasks in both ALFWorld and DABench benchmarks, showing its practical benefits.
Authors
Ji'an Lei, Jian Huang
Abstract
Agents with smaller language-model backbones are less expensive but can drift into persistent failure modes, whereas those with larger backbones are generally more reliable but more costly. This reliability-cost trade-off motivates routing methods that decide when to invoke an agent with a larger backbone: before execution, after a fixed trajectory prefix, or locally at individual steps. Our method, TACIT-SWITCH, learns permanent handoff policies from accumulated trajectory evidence and Teacher-Annotated Censored Intervention Times (TACIT). It represents each annotation as an interval-censored observation on a cumulative-risk scale. The resulting mixture-cure threshold model estimates the probability that the paired Strong rollout succeeds and, conditional on success, the handoff threshold; no teacher is required at deployment. In a mechanism-based multi-step simulation, TACIT-SWITCH improves success by 7.4-11.1 percentage points over task-level, step-level, and fixed-prefix routing baselines at comparable cost. Within that controlled simulation, ablations show that task features and cumulative trajectory risk provide complementary information. With operating points selected on development data, TACIT-SWITCH achieves the highest held-out success among learned policies on both ALFWorld (48.5% with 4B Cheap; 45.5% with 9B Cheap) and DABench (73.1%).