Power sampling improves llm outputs mostly by picking likely sequences
A Critical Look at Power Sampling when Test Time Sharpening LLMs
Machine Learning
Summary
Large language models can be improved by focusing on the most likely answers they already produce. The authors study a technique called power sampling, which aims to emphasize these likely answers more precisely. They find that common methods to do this don't fully work within reasonable computing limits, and simpler approaches can get similar benefits by just picking the best of several tries. Even when the model is fine-tuned to exactly match the target emphasis, improvements are inconsistent. This suggests the main benefit is just selecting answers the model already thinks are probable, not perfectly reweighting probabilities.
What this means in practice
- •For ai engineers: Implement simpler sample selection methods to improve language model answers without complex sampling algorithms.
- •For machine learning practitioners: Fine-tune language models targeting amplified likelihood distributions cautiously, expecting gains to be task- and model-dependent.
Authors
Abiel Malkani Talwar, Christopher Nemeth, Henry Moss
Abstract
Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution $π_α\propto π_{\text{base}}^α$ via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to $π_α$ under realistic compute budgets, and that their gains are largely recreated via simple Best-of-$N$ with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating $π_α$, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match $π_α$, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting $π_α$, but in the simpler goal of extracting sequences the base model already deems likely.