Language models improve together using adaptive instruction selection
Multi-LLM Collaborative Alignment via Stackelberg Games
Artificial IntelligenceMachine Learning
Summary
Language models learn better when they share knowledge by trying different instructions and comparing answers. The authors created a system where a leader chooses which instructions to use based on how helpful they are for learning. The models then respond, judge each other’s answers, and learn from those judgments. This way, the models get better over time by focusing on instructions that help them improve the most. Experiments show this approach beats older methods across many tasks like reasoning and coding.
What this means in practice
- •For machine learning engineers: Select and schedule tasks to train multiple language models collaboratively to improve overall quality efficiently.
- •For ai platform operators: Deploy adaptive instruction sampling in multi-model systems to boost collective model alignment and performance on diverse benchmarks.
Authors
Christina Hahn, Shangbin Feng, Dean Light, Swastik Roy, Hila Gonen, Yulia Tsvetkov
Abstract
A pool of language models can collaborate and improve collectively by learning from one another's responses. These interactions depend on the instructions used during training. Existing methods typically sample instructions uniformly, even though their usefulness may change as the models improve: an instruction on which models' responses once differed in quality may later be answered equally well, while a previously difficult instruction may begin to provide a useful learning signal. We propose Stackelberg Alignment, a game-theory-inspired leader-follower framework that turns instruction selection into an adaptive curriculum. An EXP3 bandit acts as the leader, allocating a fixed sampling budget across instructions and updating its sampling distribution using a reward that combines instruction difficulty and response discriminability. The language models act as followers: they respond to the selected instructions, evaluate one another's responses, and learn from the resulting preference signals through DPO or GRPO. The framework uses Elo-style reputation-weighted peer judgment and reputation-based opponent matching to support reliable and competitive model interactions. Experiments across three heterogeneous model pools and 12 benchmarks spanning scientific discovery, reasoning, code, instruction following, and knowledge show that Stackelberg Alignment achieves the highest macro-average across three diverse model pools, outperforming the strongest training-time baseline by up to 7.4% and the best static inference baseline by 12-25%. Analysis confirms that the adaptive leader concentrates duels on the most informative instructions, and ablations show that both reputation-weighted judgment and reputation-based matching improve the effectiveness of multi-LLM evolution.