Multi-task llm training speeds up by reusing exploration data
DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training
Machine LearningArtificial Intelligence
Summary
Training language models for many tasks usually needs lots of computing power because it involves running many training sessions repeatedly. The authors found a way to reuse old training data, called exploration histories, so the model can learn just as well without costly repeated steps. This new method, DoubleFree, makes training much faster and cheaper while keeping similar ability across different tasks. It works by using previously collected data to teach the model instead of generating new responses every time.
What this means in practice
- •For ai platform engineers: Cut multi-task language model consolidation time and GPU costs by reusing historical training data without online teacher overhead.
- •For cloud resource managers: Reduce GPU usage significantly when training specialized models across domains by applying rollout-free distillation methods.
Authors
Yingping Wang, Min Cai, Chenyu Yuan, Xi Ye, Zhijiang Guo, Jiayi Huang
Abstract
Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$-$5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$-$90.1\%$, while maintaining comparable multi-domain performance.