RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
2026-08-06 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how to improve large language models so they can solve reasoning problems in many languages, not just English. They found that focusing training on key 'reasoning pivots'—important decisions in the reasoning process—helps models transfer reasoning skills better across languages. They introduced RP-OPSD, a method that uses differences between teacher model outputs with and without English guidance to target these pivots during learning. Tests on math problems in 17 languages showed their approach works better than previous methods. Their analysis also revealed that RP-OPSD focuses more on control steps in reasoning rather than just copying text.
multilingual reasoninglarge language modelson-policy self-distillationreasoning pivotscross-lingual transfertoken-level supervisionmathematical reasoning benchmarksdistributional shiftreference anchoringsurface realization
Authors
Xinye Wang, Junxiao Liu, Shujian Huang
Abstract
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.