Latent rank collapse limits reasoning quality in implicit chain of thought models
Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning
Machine Learning
Summary
When computers use language models to solve problems step-by-step without writing down each step, their hidden reasoning steps can become too similar and repetitive. This makes the thinking less flexible and accurate. The paper's authors find this problem and create a method called WIDER, which teaches the model to keep its hidden steps more diverse during training. This helps the model think in a wider, more varied way without changing how it produces answers. Their experiments show this approach leads to better reasoning performance.
chain-of-thought reasoningimplicit reasoninglatent statesrank collapseregularizationspectral methodslanguage modelslatent spacemodel trainingrepresentation diversity
Authors
Yuwen Hao, Menglin Yang
Abstract
Chain-of-thought (CoT) reasoning improves the reasoning ability of large language models by introducing intermediate computation, but explicit rationales increase decoding length, latency, and context cost. Implicit CoT offers a more efficient alternative by moving intermediate reasoning into continuous latent states. However, latent reasoning can be unstable: successive latent states may become overly similar and collapse toward a shared dominant direction, reducing the diversity of the reasoning trajectory. In this work, we identify $\textit{latent rank collapse}$ and propose $\textbf{WIDER}$, a lightweight spectral regularizer for implicit CoT. During training, WIDER estimates the shared direction of each latent trajectory and penalizes projections onto this direction, encouraging latent states to span a broader representational subspace. The method is plug-and-play and leaves the backbone model, latent schedule, and inference-time decoding procedure unchanged. We further formulate this collapse as a geometric bottleneck in implicit reasoning, casting its mitigation as a training-time regularization problem rather than an inference-time decoding change. Extensive experiments show that WIDER improves matched implicit CoT baselines, while mechanistic analyses reveal higher effective rank, lower dominant-direction energy, and reduced redundancy among latent steps. These results highlight latent subspace utilization as an important factor for efficient continuous reasoning, providing a geometric perspective for analyzing and improving implicit CoT. Code is available at https://github.com/whitesweater/WIDER.