Best of n sampling improves agent decisions in long tasks
On Turn-Level Best-of-N Sampling for Scaling LLM Agents
Artificial Intelligence
Summary
Solving complex tasks with language models can be tricky because they need to decide what to do at each step. The authors studied a method where, at each step, one model suggests several possible moves and another model picks the best one. They found this approach helps the agent complete tasks more accurately and consistently. It also leads to smarter behavior like fewer mistakes and better following of instructions.
What this means in practice
- •For software developers: Build more reliable AI agents by selecting the best action from multiple suggestions at each decision step.
- •For robotic system integrators: Improve robot task performance by using dual-model systems to generate and evaluate possible actions before execution.
Authors
Jalaj Bhandari, Dhawal Gupta, Hongli Zhan, Zhenting Wang, Varad Pimpalkhute, Zhengzhong Liu, Eric Xing, Rupesh Srivastava
Abstract
In this work, we evaluate a simple inference-time scaling approach based on \textit{turn-level best-of-N sampling} (\method) for large language model (LLM) agents on complex, long-horizon tasks. We instantiate this using a \textit{generator-selector} framework with two LLMs; at each turn, a \emph{generator} model generates $N$ candidate actions and a \emph{selector} model selects one of them, which is then executed in the environment. This paper is closely related to the work of \cite{zainullina2025guided} who used critic models for guided search at inference time. Different from their work, we evaluate if off-the-shelf, open-weight LLMs can be effective at comparing different candidate actions directly. Experiments on long-horizon terminal tasks show that \method can lead to significant performance improvements, both in terms of accuracy (5--10\%) and consistency across repeated attempts. Analysis of agent trajectories reveal a shift in agent behaviour towards desirable patterns such as reduction in number of attempts to solve the task, improved task instruction following, debugging and testing.