Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation
Machine LearningComputation and Language
Summary
The authors studied how to make reasoning language models (RLMs) better by using extra training data with human-written answers, even when it's hard to check if the answers are right. They first improved the model with simple instruction tuning, then combined this with the original model to keep its reasoning skills. Their tests showed this method helps models perform better in tasks like coding and summarizing text without losing abilities in other areas. This approach is also very inexpensive, costing less than $3 to do.
Authors
Yu-Du Feng, Niels Mündler-Sasahara, Mark Vero, Martin Vechev
Abstract
Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement learning that drives RLM performance gains. However, training RLMs on domains that lack reliable verifiers remains challenging. Meanwhile, for both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist. In this work, we show that these data can be used efficiently to further improve RLM performance. For this, we first use classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM. Next, we merge our instruction-tuned model with the original reasoning model, recovering its reasoning behavior on the target domain. Our extensive evaluation demonstrates that our technique improves RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains. Importantly, our method is highly cost-effective, enabling such improvements for less than USD $3.