HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
2026-08-03 • Machine Learning
Machine LearningArtificial IntelligenceInformation Retrieval
AI summaryⓘ
The authors propose HindSearch, a new method to improve search-based language model agents by learning from their mistakes more effectively. Instead of just marking each try as success or failure, a fixed evaluator writes a short explanation of why a failure happened, using the correct answer as a guide. This explanation helps the agent learn better search strategies during training. Their tests showed better accuracy compared to previous methods, and they confirmed that having access to the correct answer for critiques is key to the improvement.
search-augmented language modelsreinforcement learningself-distillationhindsight learningexact-match rewardevaluation metricsearch trajectoryauxiliary losslanguage model trainingcritic model
Authors
Haowei Liu, Jiamian Wang, Hsin-Tai Wu, Zhiqiang Tao, Yi Fang
Abstract
Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge's access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement.