In-Context Density Estimation for Tabular Data
2026-08-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors developed ICED, a new model that estimates how likely different data points are in a dataset without needing to be specially trained for each new dataset. ICED is based on transformers and learns from synthetic data so it can quickly evaluate new data all at once. This lets it handle several tasks like finding unusual data, spotting out-of-place data, and creating new data, usually requiring separate tools. The authors show that ICED works as well as the best task-specific methods but needs no extra tuning or training when switching tasks.
density estimationtransformersenergy-based modelslog-densityunsupervised learninganomaly detectionout-of-distribution detectiondata augmentation
Authors
Patryk Marszałek, Jacek Tabor, Marek Śmieja
Abstract
Density estimation underlies many unsupervised tasks on tabular data such as anomaly detection, out-of-distribution detection, and data augmentation. Although all these problems reduce to questions about where probability mass lies, they are typically solved individually by fitting a separate model to each dataset, with its own hyperparameters and tuning budget. We introduce ICED, an in-context, energy-based density estimator that removes this per-dataset cost. ICED is a transformer-based model pretrained once on a synthetic prior built specifically for density estimation under an objective that fits log-density where it is informative and preserves its ordering elsewhere. In the inference, it reads a dataset as context and returns an unnormalized log-density for any query point in a single forward pass, with no fitting, sampling, or hyperparameter selection. A single frozen ICED model then drives four tasks usually handled by four specialized pipelines: density estimation, out-of-distribution detection, unsupervised anomaly detection, and generative augmentation. Across all four, it is competitive with the strongest task-specific method, while being the only approach that needs no retraining, no tuning, and no labels to move between them. The code is available at https://github.com/gmum/iced.