Model performance depends on balanced data from multiple areas during mid-training
Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training
Artificial Intelligence
Summary
This research looks at how the mix of different kinds of data used to train a computer model affects its ability to solve problems. The authors found that using a moderate amount of data from each area leads to the best results, rather than focusing too much or too little on any one area. They also discovered that later adjustments to the model cannot fully fix problems caused by an unbalanced training mix. Additionally, avoiding certain data altogether causes the model's performance to drop sharply. Overall, the findings highlight the importance of balanced data during an important training phase called mid-training.
mid-trainingpre-trainingalignmentdata compositionlogical reasoningdomain coveragefine-tuningmodel performanceSFTdata allocation
Authors
Yunpeng Xu, Kun Zheng
Abstract
Mid-training, the stage between pre-training and alignment, is where a model's per-domain data composition is typically set by data availability rather than principled design. We ask what that decision buys, and whether a later alignment pass can undo it. In a controlled logical-reasoning setting (Qwen3-8B-Base, with a 4B replication; five semantically rule-disjoint KOR-Bench domains) we train 30 allocations spanning the five-domain simplex, 24 sweep configurations plus six withheld from the fit, at five seeds each. Three findings emerge. First, every domain has an interior coverage optimum: the moderate band ($10\%$-$40\%$) is best for all five domains, and a calibrated permutation test for quadratic interiority gives $P\approx0.010$; the fitted mid-training-only curves, with 8B peaks between $9.9\%$ and $35.1\%$, reproduce for curve shape but not peak location. Second, the gaps survive a fixed-budget alignment pass: compensatory SFT raises 116/120 cells (mean $+4.32\%$) yet bridges $0/240$ pairs at a $5\%$ threshold and $30/240$ at a $10\%$ ratio, an equal-budget uniform control behaves almost identically, and a permutation null would bridge $13.8\pm3.3$ and $77.9\pm8.5$ pairs ($P<0.001$). Third, zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse is commingled with generic drift. An exploratory $θ^*$ allocation attains the largest full-pipeline gain ($+4.36\%$ vs. $+0.80\%$/$+0.64\%$\,pp) but is marginal under Welch test.