Direct Nash optimization improves language model preferences significantly
Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization
Machine LearningArtificial IntelligenceComputation and Language
Summary
Language models often struggle to match general user preferences well. The authors study a method called Direct Nash Optimization that tries to improve models by thinking of training as a game between producing good responses and selecting preferred answers. They show this method helps medium-sized models perform better compared to larger ones, using ratings from AI evaluators like GPT-4o or similar open models. Their results suggest it is possible to improve models without relying on proprietary systems for feedback, making the approach more accessible.
What this means in practice
- •For ai developers: Use Direct Nash Optimization to improve model outputs that better align with general user preferences without proprietary feedback models.
- •For chatbot platform teams: Incorporate DNO-Prct for refining medium-sized open-weight language models to provide competitive interaction quality against larger models.
Authors
Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy, Bhuvnesh Kumar, Parivesh Choudhary
Abstract
Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, and the tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate. With same-family open-weight annotators, the runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score 6 to 10 LC points below the full procedure, whereas varying the number of on-policy samples per prompt (1 to 7) moves the LC win rate by at most 2.1 points, and a margin sweep (1 to 5) run in its own pipeline by at most 2.4 points. We also report the cost of the pipeline and where the practical algorithm departs from the theory it approximates. All results are single runs, and GPT-4o is both the teacher and the evaluation judge; we discuss what this implies for the reported gains.