Synthetic data impact varies by training method in linear regression
Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
Artificial IntelligenceMachine Learning
Summary
Synthetic data can help train models when real data is limited, but sometimes it stops models from getting better, causing a performance floor. The authors analyze how synthetic data affects training in a common math problem called linear regression. They find that mixing synthetic data with real data during training causes this performance floor, but using synthetic data only at the start of training avoids it. This means whether synthetic data helps or hurts depends on how and when it's used in training, not just its quality.
What this means in practice
- •For machine learning engineers: Design training schedules that use synthetic data early to avoid performance limits in regression models.
- •For data platform teams: Manage synthetic data quality and usage protocols to maintain model performance as data scales.
Authors
Jichu li, Difan Zou
Abstract
Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Dohmatob et al., 2024), where any fixed fraction of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. In this paper, we study how synthetic data affects the generalization of one-pass SGD in high-dimensional linear regression with model shift. We establish finite-sample risk bounds for mixed and two-stage training, separating standard bias and variance from source-mismatch effects, namely fluctuation and persistent drift under mixing and filtered initialization bias under two-stage. These bounds reveal a sharp contrast: mixed training induces strong model collapse, while two-stage training avoids the floor by using synthetic data only in the first stage, showing that collapse is not inevitable under a simple data curriculum. Under a random sketch model, we further obtain scaling laws for both protocols, with tight results for mixed training in the optimization-saturated regime. These laws show that larger models may amplify synthetic-induced degradation under mixing, and quantify how high-quality synthetic pretraining may reduce bias in two-stage training. Finally, we establish an exact finite-sample necessary-and-sufficient condition for two-stage training to strictly outperform real-only training under the same real-data budget and identical real-stage updates. Overall, our results highlight that synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality and the training protocol used to incorporate it.