Cross-Relational Preference Learning for Better LLM Instruction Following
Artificial Intelligence
Summary
The authors noticed that current large language models struggle with following complicated instructions because they don't consider how different instructions relate to each other. To fix this, they created a method called Cross-Relational Preference Learning (CRPL) that builds better training data by looking at relationships between instructions and checking responses carefully. This helps the models learn to handle a wider variety of constraints in instructions. Their tests showed that CRPL improved how well models understood and followed instructions compared to older methods.
Authors
Runsheng Li, Kai Sun, Bin Shi, Bo Dong
Abstract
Large Language Models (LLMs) still exhibit limited capability in following complex instructions. While existing approaches often rely on preference learning to enhance this ability, they typically overlook the relationships between the permissible response spaces of different instructions, which restricts a model to align with subtle and diverse constraint variations. To address this, we propose Cross-Relational Preference Learning (CRPL), a novel framework for constructing preference data that explicitly models inter-instruction relationships through two key techniques: Cross-Relationship Perturbation and Cross-Region Pair Sampling. This enables the generation of more diverse preference data that captures a wide spectrum of constraint variations. Additionally, we introduce an atomic constraint-based verification mechanism to rigorously assess response satisfaction, ensuring high-quality preference pair construction. Extensive experiments across multiple preference learning methods (e.g., DPO, KTO), LLM backbones and four instruction-following benchmarks demonstrate that our approach achieves substantial improvements over prior baselines and exhibits strong generalization.