Composable decoding methods improve trade offs in language model outputs
Composable Decoding on the Probability Simplex: Theory and Implementation
Machine LearningArtificial Intelligence
Summary
Decoding in large language models decides how the AI picks the next word when generating text. The authors see decoding as a math problem where choices balance how good the model’s suggestions are against certain rules. By mixing these rules in one step, they can create new ways of decoding that naturally blend strengths of existing methods. They built a tool called CompoSimplex to test these mixes and found that combined methods can better trade off quality and variety of results than usual single strategies.
What this means in practice
- •For natural language processing engineers: Design new text generation methods that balance multiple output qualities without extra training or external scoring models.
- •For ai product developers: Build AI systems that produce more diverse and high-quality language outputs by combining decoding preferences in a single optimization step.$Commercial implications: Enables creation of commercial AI text generation tools offering better control over output quality and diversity.
Authors
Xiaotong Ji, Ahmed Khaled Khamis, Rasul Tutunov, Matthieu Zimmer, Haitham Bou-Ammar
Abstract
Decoding for large language models is typically treated as a collection of isolated sampling strategies, with limited theoretical understanding of the behaviours they induce and how their underlying objectives relate. We formulate decoding as an optimisation problem over next-token distributions on the probability simplex, balancing expected model score against regularisation under support constraints. This view recovers familiar decoding methods through choices of regularisers and support constraints; more importantly, it enables new decoders to be constructed by composing distributional preferences within a single optimisation problem without external rewards, learned critics, or model parameter updates. We introduce CompoSimplex, a library with configurable support rules, regularisation primitives, and simplex solvers for constructing and evaluating compositional decoders. We evaluate standard samplers, individual regularisers, and compositions across multiple models and reasoning tasks. Our results show that compositions can realise trade-offs between single-sample quality, multi-sample quality, and diversity that are not attained by individual decoding objectives.