AI summaryⓘ
The authors develop a method called Latent Thought Credit (LTC) to help language models improve reasoning by better understanding which parts of their 'thought process' lead to correct answers. Instead of treating answers as a single outcome, LTC breaks down reasoning into intermediate steps called latent thoughts and evaluates each by checking multiple answers generated from that step. This approach helps the model learn which thoughts are helpful and which are not, resulting in improved accuracy on math and science questions. Their tests show that LTC is better than other methods at assigning credit for good reasoning steps. They also find that checking multiple answers per thought reduces errors in evaluating these intermediate steps.
latent reasoningcredit assignmentlanguage modelschain of thoughtreward estimationhierarchical frameworkGRPOmathematical reasoningSTEM tasksadvantage weighting
Authors
Xuyang Zhao, Liting Zhang, Zichen Xu, Yong Chen, Wenjia Zeng, Shiwan Zhao, Qicheng Li
Abstract
Latent reasoning allows language models to carry out intermediate reasoning in continuous latent representations rather than fully externalizing it as discrete chains of thought. However, assigning credit to such latent thoughts from answer-only rewards is difficult: a single final answer mixes thought quality with answer-sampling noise. We propose \textbf{Latent Thought Credit (LTC)}, a hierarchical credit-assignment framework for latent reasoning. For each prompt, LTC samples multiple latent thoughts, fixes the context after each thought, and estimates thought-level expected reward by averaging rewards over multiple answers generated from that fixed context. LTC uses thought-level advantages to optimize the latent-thought phase, answer-level advantages to optimize the answer phase, and an advantage-weighted thought-matching objective that helps the policy reproduce high-credit latent thoughts. We instantiate LTC in a GRPO-style on-policy training framework and evaluate it across mathematical reasoning and STEM multiple-choice tasks. LTC achieves the best average accuracy among the compared methods, while ablations and fixed-context diagnostics show that multi-answer estimation reduces reward-estimation error and mitigates ambiguous or incorrect thought-level credit.