Continuous and discrete flow methods reveal new paths for categorical generation
Flow Duality and Source Geometry for Categorical Generation
Machine Learning
Summary
Generating categories like words or labels can be done in different ways, either smoothly or in steps. This paper shows these two ways are connected through a neat mathematical relationship involving a choice called the source distribution. The authors found that picking different starting shapes changes how the transition between categories happens and how big the vocabulary can be. Their tests hint these findings might help improve language models and early stages of category generation tasks.
What this means in practice
- •For machine learning engineers: Design categorical generation models with customized source distributions to control transition timing and vocabulary size in language tasks.
- •For natural language processing teams: Improve early-stage generative quality in language models by adjusting geometry of source input distributions during training.
Authors
Etrit Haxholli
Abstract
Continuous and discrete flow matching are usually treated as separate constructions. This paper identifies a duality between them: projecting continuous convex-interpolant paths with one-hot targets through a position-wise argmax yields discrete convex-interpolant paths. The result requires source laws with appropriate coordinate symmetry and boundary regularity, and it makes the continuous source distribution an explicit design choice for categorical generation. We derive the induced discrete interpolation behavior for Gaussian, bounded-uniform, and centered negative-exponential sources, showing that different source geometries lead to qualitatively different transition timing and vocabulary-size dependence. Small visual diagnostics and a short language-modeling pilot suggest that these source-design effects can also appear in learned transports and early generative quality.