Data-free training sharpens language model adaptation at test time
Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learning
Machine Learning
Summary
Many language AI models improve themselves by learning during testing but can start out too uncertain, making learning slower. The authors show that preparing these models beforehand without extra data can help by making their starting point more focused. This focused starting point lets the model learn more effectively during test time using only its own generated examples. They tested different ways to prepare the models and found that models with sharper initial behavior adapted best and performed better on math and reasoning tasks.
What this means in practice
- •For language model engineers: Create checkpoints prepared without external data that enable faster and more reliable adaptation on new tasks using only self-generated signals.
- •For automated reasoning system builders: Improve math and question answering capabilities by sharpening model policies before adaptation, increasing efficiency during limited learning steps.
Authors
Zhanming Zhang, Vinoth Selvendran
Abstract
Test-time reinforcement learning (TTRL) adapts language models on unlabeled test problems using supervision derived from their own samples. This makes the checkpoint's \emph{entry state} consequential: a diffuse policy provides noisier self-supervision and may spend much of a limited adaptation budget merely concentrating probability mass before reliably expressing capability it already possesses. We propose \textbf{entry-state sharpening}: use data-free training \emph{before} TTRL to prepare a general-purpose checkpoint in a state that subsequent label-free adaptation can exploit more efficiently. The idea is not tied to one training recipe; different data-free objectives can move the same base model to different entry states. Across five data-free checkpoints derived from Qwen3-4B and evaluated under an identical 15-step TTRL protocol, entry policy entropy strongly rank-orders endpoint conversion efficiency, a reliability-to-reachability measure (Spearman $ρ=-0.90$; $ρ=-0.99$ after controlling for entry reachability). The contrast across objectives is striking: R-Zero remains diffuse at $3.39$ nats and finishes below the untuned base in 6/6 matched comparisons across MATH, GPQA, and AMC, whereas SPIRAL reaches $0.07$ nats and achieves the highest post-TTRL accuracy on MATH and GPQA despite its self-play stage using no math training data. An in-domain label-free self-distillation intervention further shows that the entry state can be deliberately sharpened. These results motivate treating checkpoint preparation as a \emph{state-control problem}: use data-free training to improve TTRL readiness, with entry entropy as a label-free control signal and reachable capability as the constraint.