AI summaryⓘ
The authors study how to improve large reasoning models that work across multiple topics or domains using reinforcement learning. They find that previous methods struggle because different domains interfere with each other instead of helping. To fix this, they create a new method called Multi-domain Contrastive Policy Optimization (MCPO) that encourages the model to share useful knowledge between domains while keeping each domain’s good knowledge clear. MCPO uses a contrastive learning approach to tell the model which reasoning paths are helpful and which are not, leading to better reasoning across several domains. Their tests show MCPO works better than previous methods and even beats training models on single domains sometimes.
Large Reasoning ModelsReinforcement LearningPolicy OptimizationMulti-domain LearningCross-domain InterferenceKnowledge SharingContrastive LearningRolloutsRepresentation Space
Authors
Zongji Yu, Wenshui Luo, Yiliu Sun, Hao Fang, Runmin Cong, Chaochao Lu, Chen Gong
Abstract
Post-training has significantly enhanced the reasoning capability of Large Reasoning Models (LRMs), especially with Reinforcement Learning (RL) like Group Relative Policy Optimization (GRPO). However, GRPO-style RL methods in multi-domain settings often fail to achieve consistent improvements across all domains due to inherent interference in policy optimization. Prior studies on multi-domain RL primarily focus on alleviating cross-domain interference, while often neglecting the pivotal role of knowledge sharing, which we argue is the key to transforming cross-domain interactions from harmful competition into beneficial transfer. To address this limitation, we propose Multi-domain Contrastive Policy Optimization (MCPO), which analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing and in-domain knowledge consolidation in a contrastive manner. Specifically, for a given prompt, MCPO identifies transferable reasoning trajectories from other domains as positive examples, while treating incorrect rollouts as negative ones. It then encourages consistent representations for positive pairs and pushes negative pairs apart, thereby facilitating knowledge transfer and reducing interference. Moreover, MCPO aligns intra-domain correct rollouts to build a consolidated representation space. In this way, MCPO contrastively learns a harmonious representation space that can accommodate diverse multi-domain knowledge. Empirical results show that MCPO improves the reasoning capabilities of LRMs across multiple domains and even outperforms single-domain training in some cases. Code is available at https://github.com/Maricalce/MCPO.