Decision-Flow sampling improves reasoning in language models without retraining
Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Models
Machine Learning
Summary
Large language models can solve problems by reasoning through multiple steps, but finding the best solution path is tricky without extra training. This paper shows a way to explore many reasoning paths during model use, scoring and choosing the best complete answers instead of deciding step-by-step. The authors introduce Decision-Flow Sampling, which finds better chains of reasoning already present in the model, boosting accuracy without costly retraining. This method works well on multiple tests and models, meaning that smarter searching alone can unlock better reasoning from existing systems.
What this means in practice
- •For natural language processing engineers: Improve answer quality from large language models using a sampling method that finds better reasoning paths without extra training.
- •For ai product developers: Deploy language-model-based tools with improved reasoning accuracy without incurring the cost of reinforcement learning fine-tuning.$Commercial implications: Enables enhanced AI assistance products by improving reasoning accuracy while avoiding expensive model retraining.
Authors
Zhendong Mi, Shaoyi Huang
Abstract
A central question in LLM reasoning is whether reinforcement learning (RL) instills genuinely new capabilities or merely reshapes how existing knowledge is expressed during inference. Building on the distribution-sharpening hypothesis, which holds that RL reallocates probability mass toward high-reward trajectories already latent in base models, we ask: can we unlock those latent paths without costly RL fine-tuning? We present Decision-Flow Sampling (DF-Sample), a training-free, data-free inference-time framework that constructs a hierarchical reasoning tree, scores terminal nodes for quality, and back-propagates utilities to inform each intermediate branching decision. Unlike conventional sampling strategies that make purely local step-wise choices, DF-Sample performs explicit global trajectory evaluation before committing to a path, recovering high-quality but low-probability reasoning chains that standard decoding overlooks. On GPQA, DF-Sample achieves 45.6% accuracy, surpassing power sampling (38.9%) and GRPO (39.9%), showing that a training-free method can outperform a trained one. Across three models and four benchmarks, DF-Sample consistently outperforms baselines, indicating substantial latent reasoning potential in pretrained base models.