TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning
2026-08-04 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors propose TurnSight, a method to help large language models (LLMs) better learn how to use tools step-by-step when solving complex problems. Unlike previous methods that give feedback only after whole task attempts, TurnSight provides more detailed guidance by analyzing each step (or turn) during tool use. It does this by creating multiple "lookahead" views of possible outcomes and choosing the most reliable feedback, improving how the model learns from its own past actions. Their experiments on various benchmarks show that this approach works well.
Tool-integrated reasoningLarge language modelsReinforcement learningSelf-distillationHindsight learningTurn-level supervisionCredit assignmentCross-horizon agreement
Authors
Changle Qu, Sunhao Dai, Hengyi Cai, Yuqi Zhou, Xinran Chen, Simon, Jun Xu
Abstract
Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.