Separator tokens improve efficiency of language model reasoning
Sep-Latent: Separator Tokens as Natural Compression Anchors for Implicit Reasoning
Artificial IntelligenceComputation and Language
Summary
Complex thinking by large language models usually takes a lot of computing power and long text steps, which slows things down. The authors developed a new way called Sep-Latent that uses special marker tokens to hold important thoughts and make reasoning shorter and faster. This method adapts to how hard a question is and stops thinking when it's done, saving effort. Tests show Sep-Latent is more efficient than past short-thinking approaches and nearly matches longer step-by-step methods in accuracy. The approach also proves to be more stable when the input is changed a bit.
What this means in practice
- •For ai system engineers: Reduce computing resources needed for language models to perform complex reasoning in AI systems by using adaptive token-based compression.
- •For financial model developers: Improve speed and stability of automated reasoning in financial prediction models by compressing logic into flexible latent tokens.
Authors
Yinlong Xu, Jianfeng Huang, Zhengdong Zhou, Mingjie Lin, Fuyu Lv, Dan Ou, Hongxia Xu, Jian Wu
Abstract
Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning through explicit Chain-of-Thought (CoT), but the associated computational overhead and token verbosity remain significant barriers to efficiency. While implicit reasoning in continuous latent spaces offers a promising alternative, existing methods like Coconut and CODI often rely on fixed-length latent sequences and suffer from representational collapse. In this paper, we propose Sep-Latent, an adaptive implicit reasoning framework that utilizes separator tokens as structural semantic anchors. By introducing an Attention-Decay mechanism, we force the model to aggregate and compress reasoning logic into these anchors during training. We then align explicit and implicit pathways through multi-level distillation across all layers to ensure logical consistency. Crucially, Sep-Latent enables dynamic reasoning depth by adaptively terminating the latent thinking phase based on query complexity. Experimental results on the GSM8k-Aug benchmark show that Sep-Latent achieves an accuracy of 56.8\% with an average of only 5.4 tokens, significantly outperforming prior fixed-length implicit models in efficiency while narrowing the performance gap with explicit CoT. Further analysis via PCA and symbolic variant testing confirms that our method ensures semantic grounding and exhibits superior robustness to perturbations.