Selection rules influence when adaptive halting improves looped language models
When Does Adaptive Halting Win? Selection Rules Matter in Looped Language Models
Machine LearningComputation and Language
Summary
This paper looks at how different ways of deciding when a language model should stop thinking can affect its performance. The authors found that the choice of the stopping rule can change whether one method outperforms another. They tested this on a certain 1.4 billion parameter model and saw that some methods saved more computation but sometimes at the cost of accuracy. They also show that simple baseline methods can perform as well as more complex learned methods, especially when tested on data the model hasn't seen before. Overall, they suggest comparing stopping methods carefully with strong baselines and on diverse tests.
What this means in practice
- •For ai platform engineers: Optimize resource use by choosing stopping rules that save computation in looped language models without large accuracy loss.
- •For chatbot system builders: Improve response efficiency in chatbots by applying adaptive halting with careful stopping rule selection and out-of-distribution testing.
Authors
Mahesh Reddy Pagadala
Abstract
Looped language models can stop iterating early, and recent work shows that post-hoc readouts of the loop trajectory often rival the learned exit gate of the looped model Ouro. Such comparisons reduce each method to one operating point chosen by a selection rule. We ask whether that rule decides the outcome. On a frozen Ouro-1.4B with four loops, we compared a trained stability probe ($π_1$-x) with Ouro's native gate, adapted to a per-question stopping decision, in pre-registered studies; the confirmatory study was designed after an exploratory finding on development data and evaluated once on a sealed test set of 8,517 questions. In the pre-registered confirmatory comparison, which selected operating points with a bootstrap-screened 1 pp accuracy-loss rule, the probe saved more of the four-loop budget than the native gate (+0.053 gross macro saving over three benchmarks, 95% CI [+0.049, +0.057]), mostly from CommonsenseQA, where the probe's test upper bound on accuracy loss exceeded 1 pp. Under a point-only rule, a pre-specified secondary analysis on the same test set reversed the preference (-0.022 [-0.026, -0.018]), with operating points that exceeded the budget. In pre-specified secondary comparisons, the probe showed no demonstrated advantage over an untrained KL divergence between successive loops (+0.00125 [-0.0041, +0.0065]); we emphasise this result post hoc and claim no equivalence. The probe also broke its budget on one of two out-of-distribution benchmarks (3.26 pp loss on Social IQa). Savings are estimated from stored full-depth traces, not measured inference time, and our screen carries no formal risk-control guarantee. We conclude that halting methods should be compared as signals together with selection procedures, with strong untrained baselines and out-of-distribution checks.