Fine-tune large language models efficiently using noisy predictions

Ambient LLMs: Data-Efficient Supervised Fine-tuning via Temperature Matching

Machine LearningArtificial IntelligenceComputation and Language

Summary

It can be hard to train large language models with small amounts of new data because they forget what they learned before or overfit the new data. The authors introduce a way to fine-tune these models more carefully by having the updated model try to imitate a frozen version of itself, but in a softer, less exact way. This method uses a 'temperature' setting to adjust how closely the new model follows the old one’s predictions. Their approach improves performance in adapting models to new tasks and instructions while avoiding forgetting.

What this means in practice

  • For machine learning engineers: Improve fine-tuning of language models on small datasets by preventing overfitting and forgetting through temperature-controlled matching with frozen models.
  • For software developers integrating ai assistants: Enhance the adaptation of language models to specific instructions and domains by using ambient supervised fine-tuning for better reasoning and response quality.

Authors

Giannis Daras, Marios Papachristou, Adam Klivans, Antonio Torralba, Constantinos Daskalakis

Abstract

Collecting large-scale, diverse, and high-quality post-training datasets for autoregressive language models is challenging. Consequently, Supervised Fine-Tuning (SFT) is typically applied to datasets significantly smaller than those used for pre-training, which leads to overfitting and the catastrophic forgetting of the tails of the data distribution. To prevent this collapse, we introduce Ambient SFT, a principled framework for leveraging imperfect predictions to augment the fine-tuning set. In our method, a "frozen" version of the pre-trained model serves as a noisy teacher that the fine-tuned model must match at a coarse level. In particular, we require that the high-temperature versions of the conditional distributions produced by the post-training model should match the high-temperature versions of the pre-training ones. The temperature hyperparameter controls the level of trust in the pre-trained model, with the limit $T\to \infty$ reverting back to regular SFT and $T=1$ to knowledge distillation. We demonstrate that our approach outperforms standard SFT across domain adaptation, instruction-tuning, and reasoning, leading for up to $21\%$ average relative improvement to standard SFT.