Language models update continually using label-free gradient matching
Online Label-Free Gradient Matching for Continual Language Model Training
Artificial IntelligenceComputation and Language
Summary
Language models need constant updating as the data they learn from changes over time. The authors present OLGA, a method that updates models using a small set of unlabeled examples created by matching gradients from the model’s own predictions. This lets the model keep learning efficiently without needing full, labeled data or long retraining periods. OLGA also works with data that arrives continuously and runs faster on large datasets compared to existing methods.
What this means in practice
- •For machine learning engineers: Update large language models efficiently with small unlabeled example batches in live customer-facing systems.
- •For data pipeline developers: Combine gradient-based distillation with sample selection to speed up training on growing text datasets in production.
- •For automated customer support teams: Improve real-time language model updates to maintain accuracy as customer queries and topics evolve.$Commercial implications: Enables more accurate and timely updates to language models powering customer support chatbots, improving service quality.
Authors
Sajjad Pakdamansavoji, Mohammad Reza Rezaei, Ilija Subasic, Pedro Garzon, Radu Paradovschi, Zhao Chen
Abstract
Language models in customer-facing products must be continually updated, since the data they serve shifts over time. As retraining on the full accumulated data is impractical, each update should instead run on a small set of examples carrying the training signal of all data seen so far. The strongest current methods build this set by gradient matching, which synthesizes examples whose training gradients match those of the full data. However, they require ground-truth labels, run once on a fixed snapshot rather than on a growing stream, and are slow on large data. We introduce \textbf{OLGA}, which lifts these limitations. It distills unlabeled data by drawing the gradient matching targets from the model's own next-token predictions. Across five text classification benchmarks with phi-1.5, it improves accuracy by 10 percentage points over the strongest selection baseline using as few as five examples. OLGA also distills online data streams by keeping a running rehearsal memory of synthetic examples, which is re-distilled together with each new batch as it arrives. In the online setting, it outperforms the strongest selection-based continual baseline by 6.6 stream-AUC percentage points. Finally, our method scales to large data by pairing distillation with selection in a hybrid pipeline, letting selection handle the mild stage of compression and distillation the extreme one. It cuts runtime by 30\% versus distillation alone at similar accuracy, and beats the best selection-only pipeline by up to 12.6 percentage points.