Midtraining improves base ai models for software tasks and agent training
Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Training large AI models in stages can affect how well they learn to do tasks. The authors show that adding a special 'midtraining' step between early training and later fine-tuning helps models better handle complex software engineering tasks and makes them more robust for agent-based training. This midtraining involves feeding models a mix of real software data and some agent interaction data at very long context lengths. Experiments on different model sizes showed clear improvements in software benchmarks and robustness after midtraining.
What this means in practice
- •For ai model trainers: Increase software task accuracy and agent training robustness by integrating a midtraining stage with diverse code data and agent-like interactions.
- •For software engineering teams: Utilize models improved by midtraining to better understand and automate complex software repository and development data analysis.
Authors
Soumye Singhal, Joseph Jennings, Ka Shun Shum, Alexander Bukharin, Khushi Bhardwaj, Zirui Liu, Venkat Srinivasan, Jonathan Li, Yash Jain, Kezhi Kong, Markus Kliegl, Olivier Delalleau, Sanjeev Satheesh, Mostofa Patwary
Abstract
Frontier models, general-purpose and domain-specific alike, now include a midtraining stage between pretraining and post-training, but little has been published about how this stage is built. We present an open midtraining recipe that primes base models for agentic post-training, and we show its effectiveness on two Nemotron models of different sizes. Pretraining typically runs at short context and sees almost no agent interactions, while SFT data is mostly synthetic, and very long SFT stages can hurt the model's later RL trainability. Our midtraining stage sits between the two. It trains at 256K context for hundreds of billions of tokens on a diverse mixture of natural software-engineering data like packed repositories, pull requests, commit histories, and build logs together with a small share of agentic traces. On Nemotron 3.5 Lightning, 500B midtraining tokens improve SWE-bench Verified from 60.7 to 64.5, SWE-bench Multilingual from 55.8 to 62.8, and SWE-bench Pro from 32.0 to 35.5 after identical full SFT. On the larger Nemotron 3.5 Super, 280B midtraining tokens improve DeepSWE, a harder benchmark of long-horizon tasks, from 11.7 to 15.7 and SWE-bench Verified from 72.9 to 74.3 after identical full SFT. Midtraining also improves robustness across agent harnesses on both models. We report the ablations behind the recipe and an evaluation protocol built on short fixed SFT runs, since base-model proxy metrics did not correlate reliably with post-training results.