Language Chain in Alignment: Cross-Lingual Ranking Preference Optimization
2026-08-24 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors address the problem that large language models tend to perform worse in languages other than English because training data is mostly English-based. They introduce a new method called CRPO, which uses preference data from English to help models learn better in other languages by comparing and ranking responses within and between languages. Their approach improves how well models follow instructions and use knowledge in five different languages. The results show that CRPO makes preference learning more stable and outputs better responses across languages.
Large Language ModelsPreference LearningCross-Lingual AlignmentLambdaLossInstruction FollowingKnowledge UtilizationMultilingual NLPRanking OptimizationParallel Preference PairsLanguage Adaptation
Authors
Seungyoon Lee, Minhyuk Kim, Jungseob Lee, Heuiseok Lim
Abstract
The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-Lingual Ranking Preference Optimization (CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment.