OrderGPT learns best generation sequence for structured data
OrderGPT: Generation Order as a Learnable Modeling Choice
Artificial IntelligenceMachine Learning
Summary
Generating sequences like text or human poses usually follows a fixed order left to right or random. This paper shows that picking a better order can improve how well models understand and create data. The authors introduce OrderGPT, which learns the best order to generate parts of the data by observing its own attention patterns during training. This approach improves results across languages, human poses, and table data when compared to fixed orders.
What this means in practice
- •For language model developers: Improve natural language generation by adapting generation order to better capture token dependencies beyond left-to-right sequences.
- •For robotics and animation engineers: Generate human pose sequences more naturally by learning anatomically consistent ordering of joints during modeling.
Authors
Chenhe Yang, Yuhuan Lyu, Tianyang Hu
Abstract
Generative modeling of discrete data has two popular paradigms: autoregressive models such as GPT, which factorize a sequence in a left-to-right (L2R) order, and masked diffusion models, which are essentially any-order autoregressive modeling. Rather than competing methods, these are the two extreme points of a single axis -- the generation order. For structured discrete domains without a natural traversal, both extremes may be suboptimal, yet the intermediate choice -- a single well-adapted order -- remains largely unexplored. We introduce OrderGPT, which builds on Any-Order GPT to learn a task-level generation order without canonical-order supervision. Our key observation is that predicting variables from randomly sampled reveal prefixes produces recurring attention patterns in the backbone that provide a model-dependent signal of predictive dependencies. We aggregate this signal into a reciprocal dependency graph and use a spectral teacher to derive continuous generation priorities. A lightweight OrderMLP integrated into the GPT backbone learns to predict these priorities from attention, incorporating order learning into the architecture within a single training run. Training gradually shifts toward the learned order, which is then frozen for subsequent autoregressive training and generation. The same mechanism adapts across modalities: adaptively it recovers near-L2R dependency structure in language, learns anatomically organized factorizations on human pose, and discovers a factorization on tabular data that places statistically and semantically related fields nearby, while improving downstream modeling metrics across all domains. The code is available at https://github.com/Chenhe-yang/OrderGPT/.