Low precision pretraining slows learning of rare facts in transformers
What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
Machine Learning
Summary
When training AI models with low-precision math operations, they seem to perform similarly based on usual tests like validation loss. But this study found that these models actually learn rare facts more slowly compared to full-precision models. The researchers showed this by training models with 4-bit math and comparing how many examples they needed to remember rare information. They also explored how different types of noise affect training efficiency and concluded that loss alone doesn’t tell the full story about learning performance.
What this means in practice
- •For machine learning engineers: Improve model training pipelines by measuring rare fact learning, not just validation loss, when using low-precision arithmetic.
- •For ai hardware developers: Design precision formats and quantization techniques optimized for preserving rare fact acquisition efficiency during transformer pretraining.
Authors
Chang Lu, Mengjiao Zhang
Abstract
Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32$\times$ and 1.52$\times$ as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26$\times$ and 1.30$\times$ when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2$\times$ at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.