Post-training improves accuracy but reduces solution variety in language models
Sharpening Tax in Post-Training
Artificial IntelligenceMachine Learning
Summary
Sometimes, training large language models after their initial training helps them get better at giving one clear answer but makes them less likely to come up with different possible answers. The authors found that the original models, with a bit of extra help during use, can find more solutions than the post-trained models, although usually less accurately on the first try. They created a new way to measure how much variety is lost after this extra training and found that the loss is common. They also designed a new sampling method that helps keep more solution variety while still improving accuracy.
What this means in practice
- •For machine learning engineers: Improve multi-turn AI agents by balancing answer accuracy with diverse solution generation during model training.
- •For software developers using ai tools: Use adaptive sampling methods to get more diverse outputs from language models in coding and reasoning tasks.
Authors
Changdae Oh, Qi Zeng, Qi Qi, Andrey Zhmoginov, Deren Lei, Yun He, Hoang Phan, Hangoo Kang, Azalia Mirhoseini, Sharon Li
Abstract
An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.