Model training performs best with moderate data from each domain
Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training
Artificial Intelligence
Summary
Training AI models on data from multiple areas is tricky because too much or too little from any one area can hurt performance. The authors found that using a moderate amount—between 10% and 40%—of data from each domain leads to the best results. They also found that trying to fix imbalances later in training doesn’t fully solve the problem, and skipping some domains entirely causes the model’s accuracy to drop sharply. This shows that how data is balanced during mid-training really matters.
What this means in practice
- •For machine learning engineers: Set up mid-training data mixtures in multi-domain models to balance domain coverage for better reasoning accuracy.
- •For ai product managers: Plan training data strategies that avoid overusing any single domain, improving ultimate model performance in multi-domain applications.
Authors
Yunpeng Xu, Kun Zheng
Abstract
Mid-training, the stage between pre-training and alignment, is where a model's per-domain data composition is typically set by data availability rather than principled design. We ask what that decision buys, and whether a later alignment pass can undo it. In a controlled logical-reasoning setting (Qwen3-8B-Base, with a 4B replication; five semantically rule-disjoint KOR-Bench domains) we train 30 allocations spanning the five-domain simplex, 24 sweep configurations plus six withheld from the fit, at five seeds each. Three findings emerge. First, every domain has an interior coverage optimum: the moderate band ($10\%$-$40\%$) is best for all five domains, and a calibrated permutation test for quadratic interiority gives $P\approx0.010$; the fitted mid-training-only curves, with 8B peaks between $9.9\%$ and $35.1\%$, reproduce for curve shape but not peak location. Second, the gaps survive a fixed-budget alignment pass: compensatory SFT raises 116/120 cells (mean $+4.32\%$) yet bridges $0/240$ pairs at a $5\%$ threshold and $30/240$ at a $10\%$ ratio, an equal-budget uniform control behaves almost identically, and a permutation null would bridge $13.8\pm3.3$ and $77.9\pm8.5$ pairs ($P<0.001$). Third, zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse is commingled with generic drift. An exploratory $θ^*$ allocation attains the largest full-pipeline gain ($+4.36\%$ vs. $+0.80\%$/$+0.64\%$\,pp) but is marginal under Welch test.