Continuous diffusion models improve conditional generation for mixed data
DiffLTM: Continuous Diffusions Large Tabular Models for Conditional Generation
Machine Learning
Summary
Generating realistic data based on some given conditions is important but tricky when data has both numbers and categories, especially if some categories are rare. The paper presents a new way to create such data using continuous diffusion models that handle categories better, supported by a new theory for categorical loss. They also introduce new measures that focus on rare groups and expand what conditions can be used. Their model works well on labor data and can reflect known economic patterns.
What this means in practice
- •For labor economists: Use the model to simulate labor market data that reflects complex real-world patterns for analysis and policy testing.
- •For data science teams: Generate realistic conditional synthetic data with mixed types and rare categories to enhance model training and testing.
Tested on one dataset.
Authors
Alankrita Bhatt, Mukur Gupta, Andrea Montanari, Pulkit Tandon
Abstract
Conditional generation is a key capability to be supported by Large Tabular Models, enabling a number of applications. Accurate conditional generation has to contend with two key challenges: $(i)$ Data comprise a mixture of numerical and categorical entries, the latter often with high cardinality and heavily unbalanced proportions; $(ii)$ Conditional generation becomes increasingly difficult for rare sub-populations. The best models in this class use a hybrid of continuous and discrete diffusions. In this paper, we design a continuous diffusion model with strong generation quality. An important element is the use of cross-entropy loss for categoricals, for which we provide a first theoretical justification. We introduce metrics that focus on rare subpopulations. Further, we generalize conditional generation techniques to enable new types of conditioning. Finally, we show that our model (trained on labor data) can capture classical insights from econometrics. Code is available at https://github.com/granica-ai/DiffLTM