Stable training method improves large language model fine-tuning accuracy
BRAT: Bounded Residual-Aligned Training for Stable Multi-Pass LLM Post-Training
Machine Learning
Summary
Training large language models using reinforcement learning can be tricky because reusing the same training samples causes the model to overfit and behave unpredictably. The authors found that repeated training on the same samples reduces the quality of updates and leads to unstable results. To fix this, they created a new method called BRAT that adjusts training steps carefully to avoid drifting too far from the original samples and controls the feedback signals. Their approach achieves more stable training and better accuracy on tests, even when using multiple passes of the same data.
What this means in practice
- •For machine learning engineers: Improve fine-tuning stability and accuracy when optimizing large language models with reinforcement learning reuse strategies.
- •For ai platform developers: Implement bounded gradient weighting to prevent training collapse during multi-pass language model updates.
Authors
Xunlan Zhou, Yuzhe Zhao, Xuhong Huang, Feng Chen, Xuanlin Chen, Lang Feng, Bo An
Abstract
Reinforcement learning for large language model post-training often reuses each batch of sampled rollouts for several gradient passes to amortize the cost of generation. Under such multi-pass reuse, existing objectives fail in different ways: unclipped updates collapse as the policy drifts away from the sampling policy, adding a negative baseline avoids collapse but reinforces every sampled response, and PPO-style ratio clipping cuts off gradients abruptly. We first ask why reuse is fragile. Measuring how well an update estimated from a small rollout group aligns with the KL-regularized target on held-out rollouts of the same prompts, we find that this alignment drops by nearly an order of magnitude from in-sample to held-out rollouts across three base models, suggesting that repeated passes largely overfit the sampled group. Motivated by this finding, we propose Bounded Residual-Aligned Training (BRAT), motivated by a proximal update toward a bounded reweighting of the sampling policy. BRAT assigns each token a gradient weight that decreases linearly with its probability ratio and vanishes at a trust-region boundary, and normalizes group-centered rewards by their maximum magnitude, keeping advantages zero-mean and bounded without extra hyperparameters. With four reuse passes, BRAT avoids training collapse, limits policy drift, and achieves higher average accuracy with lower seed variance on most benchmarks.