Generative models and human preferences change together over time
Self-Consuming Generative Models with Co-Evolving Human Preferences
Machine Learning
Summary
When people pick their favorite model-generated examples to retrain AI models, their tastes often shift because of what the models produce. The authors studied how this back-and-forth changes both the AI’s outputs and what people like over many rounds. They found that without outside real examples, the model tends to get stuck favoring just one type of output, amplifying initial biases. But mixing in some real reference data during training helps keep the model balanced and predictable. They also created a method to pick the right mix of real data to guide the process toward better outcomes while saving effort.
What this means in practice
- •For machine learning engineers: Improve generative model training by mixing curated synthetic and real reference data to prevent bias amplification and achieve balanced outputs.
- •For content platform operators: Better manage user preference shifts in recommendation or generation systems by controlling reference data injection to sustain diverse and desirable content.
Authors
Xiukun Wei, Tian Xie, Ding Zhu, Xueru Zhang
Abstract
Generative models are increasingly trained in self-consuming iterative loops, where users curate preferred samples from model-generated candidates and the curated samples are used to train future generations of the model. Prior work has largely assumed fixed user preferences, but in practice exposure to model outputs gradually reshapes what users perceive as desirable, creating a feedback loop in which model distributions and user preferences co-evolve. We take a first step toward understanding the long-term behavior of such coupled dynamics. We show that when training relies entirely on user-curated synthetic data, iterative curation amplifies initial biases and drives the system toward one of multiple singleton equilibria in which the instance holding an initial advantage eventually dominates. In contrast, injecting reference data into training at a sufficiently large rate fundamentally changes the dynamics and yields a unique globally attracting equilibrium. Building on this insight, we study how reference-data injection can be used to control long-term outcomes, and propose an efficient algorithm that jointly selects a reference distribution and its mixing weight to steer the coupled system toward equilibria that preserve desired attributes while minimizing data collection costs.