QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents
2026-06-30 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors explain that when AI agents take many actions over long tasks, it’s hard to learn well if feedback only comes at the end. To improve this, people try giving feedback on each step, but testing these methods is complicated and expensive. The authors created QVal, a way to measure how good these step-by-step feedback signals are without needing full training, by checking if the signals rank actions correctly based on a strong reference policy. Using QVal, they tested 21 methods and found that simple approaches often work better than recent complex ones, with results consistent across different models and tasks.
LLM agentslong horizon tasksdense supervisionQ-valuesreference policytraining-free evaluationstepwise feedbackprompting baselinesbenchmarkingmodel backbones
Authors
Sergio Hernández-Gutiérrez, Matteo Merler, Ilze Amanda Auzina, Joschka Strüber, Ameya Prabhu, Matthias Bethge
Abstract
LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-only rewards provide too sparse guidance, failing to inform the model about the goodness of intermediate actions. Dense supervision methods aim to solve this problem by scoring intermediate steps, from intrinsic confidence to self-distillation and embedding similarities. However, it is common practice to evaluate them by measuring the downstream performance of a training pipeline that integrates them. This is expensive, conflates supervision quality with training engineering confounders, and renders different methodological families requiring distinct training setups incomparable. As a result, dense supervision methods are rarely benchmarked on common ground. We introduce QVal, a training-free testbed for directly evaluating dense supervision signals. Given a state-action pair, QVal measures how well a method's score is Q-aligned: whether it orders actions according to the Q-values of a strong reference-policy. This lets us compare signals before any training run and separate signal quality from other engineering choices. We instantiate QVal as QVal-v1.0, benchmarking 21 dense supervision methods across four diverse environments and seven methodological families, with over 1.2K evaluation experiments across six open-weight model backbones. We find that simple prompting baselines consistently outperform recent dense supervision methods from the literature, and that performance clusters strongly by family. These findings hold across model sizes, environments, and observation modalities. QVal is designed to be easily extensible to new environments and methods, enabling researchers to iterate on dense supervision methods before any training run.