Inference-time scaffolds boost model power beyond training alone

Understanding the Benefits of Scaffolds through Computational Separations

Machine LearningArtificial Intelligence

Summary

The paper studies how using extra procedures called scaffolds during a model's reasoning can greatly improve what the model can do. These scaffolds call the model multiple times and keep track of information between calls. The authors find that training a single model to imitate this scaffold-based approach cannot match its performance because the scaffold effectively performs more complex computations. They also use theoretical proofs to explain this limitation and confirm with experiments that making the model bigger or simplifying the task helps close the gap. This means running these scaffold procedures at inference time gives models a kind of thinking power that training alone cannot easily replicate.

What this means in practice

Authors

Jingchu Gai, Adam Block, Andrej Risteski

Abstract

Inference-time harnesses, i.e., external procedures that call a model many times and keep state across calls, can substantially improve model capabilities. In this paper, we investigate the benefits of using such harnesses and the extent to which these benefits can be internalized through training. More precisely, we conceptualize a harness $\mathcal{A}$ as an arbitrary wrapper around a base model $M$ and attempt to distill a student model from the harness's traces. Empirically we find that, even with simple scaffolds and small models, the inference-time harness significantly outperforms a distilled student. We then prove that such a gap is due to a fundamental representational limitation: inference-time harnesses can compute a closure of their per-call primitive that is $\NL$-complete even when the primitive is a trivial $\AC^0$ lookup, and that no fixed-depth forward pass can represent under the widely believed conjecture that $\NL\not\subseteq\TC^0$. Our theory suggests that expanding the representational capacity of the student relative to the task at hand, e.g., by enlarging the model, making the task easier, or lengthening its chain-of-thought, is necessary to close the gap, which we verify empirically. Finally, we scale to an agentic harness on competition math, where the same gap reappears. Overall, our results show that a harness confers on the base model a profound gain in representational power, one that training a same-class student on its traces cannot transfer back into the weights.
View PDFOpen arXiv