Small language models improve reasoning using offline large model feedback
Offline Guidance, Online Reasoning: Reusing LLM Feedback for Small Language Models
Computation and Language
Summary
Large language models can solve tricky reasoning problems but are expensive to use all the time. Smaller models are easier to run but not as good at reasoning. The authors created a way to use advice from a large model once and save it, so the smaller model can use that advice repeatedly later for similar problems. This approach helps small models reason better without needing extra training or more calls to the large model.
What this means in practice
- •For software developers: Enable small language models on user devices to solve reasoning tasks better by reusing offline corrections from large models without internet calls.
- •For enterprise ai teams: Improve deployment efficiency by enhancing on-premise language models’ reasoning through stored guidance from prior large model interactions.
Authors
Bohan Zhang, Linan Yue, Weibo Gao, Pengyu Chen, Hong Guo, Yanqi Hao
Abstract
Large language models (LLMs) offer strong reasoning capabilities but are often costly to access through commercial APIs, while small language models (SLMs) are easier to deploy locally yet remain weaker in reasoning. This capability-deployment gap has motivated LLM-SLM collaboration, which aims to improve SLM reasoning using LLM capabilities while preserving the deployment advantages of SLMs. Existing approaches mainly follow two paradigms. Knowledge distillation uses LLM-generated answers and reasoning trajectories to train SLMs offline, but requires parameter updates and additional training. Alternatively, online collaboration routes difficult problems to an LLM or leverages LLM-generated guidance and corrections when an SLM encounters difficulties. Although effective, online collaboration requires repeated LLM access. Moreover, the guidance produced for a particular problem is discarded after inference and cannot benefit subsequent problems involving similar reasoning states. In the paper, we focus on a more constrained setting in which the LLM is accessed only offline, the SLM parameters remain fixed, and online inference is performed solely by the SLM. To this end, we propose Reusable Latent Correction (RLC), which converts one-off natural-language guidance from a black-box LLM into persistent corrective experiences in the hidden space of an SLM. RLC stores these experiences in an external bank and retrieves them according to the SLM's current reasoning state, enabling the SLM to reuse LLM-derived corrections during inference without any online LLM calls. Experiments across multiple reasoning benchmarks and SLM scales show that RLC consistently improves SLM reasoning without parameter updates or online LLM calls. Code is available at https://github.com/ZBH031/reusable-latent-correction.