New algorithm improves language model alignment using uniform information density
Uniform Information Density-Based Preference Optimization
Computation and Language
Summary
Humans tend to prefer language where information is spread out evenly, which makes understanding easier. These researchers used that idea to create a new method for training language models to produce text people like more. Their method measures how evenly information is distributed and uses that to improve the model's outputs. Tests show it performs as well as or better than previous methods. This helps make AI language systems sound more natural and easier to understand.
What this means in practice
- •For ai developers: Enhance AI text generation systems to produce outputs that align better with human preferences by optimizing for uniform information distribution.
- •For language technology engineers: Improve natural language generation tools by incorporating measures of information uniformity to produce more cognitively efficient text.
Authors
Yongyuan Li, Yang Xu
Abstract
To enable large language models to generate outputs that align with human preferences, optimization algorithms based on the comparison between chosen and rejected output pairs have been developed and been proven effective. Some recent endeavors have utilized certain linguistically-derived assumptions about the preference pairs to guide the design of more precise optimization directions. However, human preferences are complex and hard to measure by simplistic criteria, and it is important to incorporate broader and more sophisticated insights about preferred vs. not-preferred language. In this study, we adopt a well-established psycholinguistic theory, Uniform Information Density, to develop a novel preference optimization algorithm, UIDPO. The basic assumption is that humans prefer to produce language in which information is uniformly distributed, because uniformity induces less cognitive processing effort. To implement this idea into the algorithm, we derive a compound metric for information uniformity by combining the global standard deviation and the local second-order difference, and integrate it to the objective of preference optimization. We compare UIDPO with SimPO, a previous state-of-the-art method, obtaining comparable and sometimes better performance on AlpacaEval2, ArenaHard, and MT-Bench. Additional linguistic analysis on complement clauses also shows that UIDPO can better reflect the tendencies of keeping information uniform in natural language than baselines. Our results show that the subtle human preference over the information rate in language is a helpful source for achieving better human alignment for LLMs. The code repository is published here: https://github.com/CLCS-SUSTech/UIDPO.