Llms improve tool use by comparing next possible actions first
Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents
Artificial Intelligence
Summary
Large language models (LLMs) perform complex tasks by using many steps, each involving picking a tool to use. The authors found it helpful if the LLM estimates how good each possible next tool is before actually using it, rather than just rolling forward blindly. They created a method called CITA that trains a model to compare possible next tool choices based on past behavior, simulations, and language model judgments. This approach helps the LLM make better step-by-step decisions and improves success on tasks needing many tool steps.
What this means in practice
- •For software developers integrating llms: Build systems that improve multi-step tool use by having the model estimate and compare possible next actions before executing them.
- •For automation engineers: Enhance automated workflows relying on language models by embedding value estimates of intermediate tool choices to increase success rates in complex tasks.
Authors
Yu Li, Zheng Zhang, Xin Liu, Shengtian Yang, Guangfeng Cai, Lei Feng
Abstract
Large language models (LLMs) rely on long-horizon tool invocation sequences for complex tasks, where each invocation can alter the task state and condition subsequent decisions. In long-horizon tool use, final-outcome rewards provide weak credit assignment over long interaction traces. Step-level rewards can offer more targeted feedback, but obtaining reliable step supervision often requires human or LLM judgment, or additional rollouts to estimate the downstream effect of an intermediate decision. In this paper, we argue that effective tool-use agents should estimate the long-horizon value of a possible next tool invocation before executing it. This objective requires comparative supervision over alternative invocations under the same context, while logged trajectories only contain the invocation that was actually taken. Therefore, we propose Comparative Inference for Tool-use Agents (CITA). CITA trains a Comparative Inference Model (CIM) from paired signals that combine observed tool behavior, scalable supervision from a Bayesian tool-graph simulator, and semantic judgments from LLM-based comparison. The resulting CIM learns to estimate how likely a possible next tool invocation is to support final task success under the current context. Across three tool-use benchmarks and multiple backbone LLMs, CITA consistently improves Tool F1 and task success. Additional analysis shows that CIM learns accurate step-level value estimates for comparative tool choices.