Active critic learning improves long-horizon language model agents
FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
Machine Learning
Summary
Long tasks with many steps make it hard for AI models to understand which early actions helped reach a good final result. The authors introduce FOCUS, a new way to teach a part of the AI (called the critic) to better estimate the value of intermediate steps by actively choosing which parts of the task to practice and learn from. This approach helps the AI assign credit more accurately to earlier actions in a task. Tests show that FOCUS does better than other methods on tasks like legal reasoning, code debugging, and deep search.
What this means in practice
- •For legal technology developers: Improve AI tools for complex legal reasoning with better credit assignment in long task sequences.
- •For software engineering teams: Enhance code debugging assistants using refined value estimates for long sequences of code analysis.
Authors
Zhengping Jiang, William Jurayj, Liaoyaqi Wang, Chunsheng Zuo, Anqi Liu, Benjamin Van Durme
Abstract
Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a trajectory incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL method that uses selective branching to help the critic network learn its value estimator. The rollout controller facilitates active learning for the critic by sampling continuations from states that are expected to most reduce the critic's estimation error. These refined value estimates then score the segment between consecutive states with a TD($λ$) advantage, while the sampled evidence supplies training targets for the critic. FOCUS supports long-horizon agentic reinforcement learning through a modular context-management interface that decouples context management from rollout allocation. Our experiments show that using compaction-induced segment boundaries for value estimation and credit assignment, FOCUS outperforms GRPO and VinePPO with matched per-step generation budgets on legal reasoning, deep search, and code localization benchmarks, improving held-out accuracy at Qwen3.5-9B scale by 7.2 points over the strongest baseline on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs. 52.4).