Dataset distillation sets do not simply add up when combined
The Composition Gap in Dataset Distillation
Machine Learning
Summary
Sometimes, people shrink big training data into small, fake datasets to save time and space. This paper shows that when different groups do this shrinking separately and then combine their small datasets, it doesn't work like just merging the original big data. The reason is that the process changes how data behaves in ways that don’t add up nicely. The authors found exactly why and by how much this combination can fail, especially for some types of problems. So, just testing small datasets alone isn't enough to know if combining them will work well.
What this means in practice
- •For machine learning engineers: Evaluate the quality of combined distilled datasets to improve federated learning model training accuracy.
- •For data platform developers: Design data compression workflows that avoid pitfalls when merging synthetic datasets from multiple data owners.
Authors
Guang Li, Takahiro Ogawa, Miki Haseyama
Abstract
Dataset distillation compresses a training set into a small synthetic set, usually evaluated one at a time. In federated and data-governance settings, several parties distill their own data and a user trains on their union. We ask whether the union of separately distilled sets reproduces training on the union of the real data composability and show that it can fail even when every source is distilled exactly and the total budget admits an exact joint distillate. Compressing a training trajectory into fewer steps transforms the source statistics nonlinearly, so averaging compressed sources differs from compressing their average. For quadratic objectives we derive the exact composition error for two-to-one step compression in terms of the source-Hessian variance and the linear terms of the losses, and on a smooth network at small step sizes this prediction captures the local endpoint discrepancy in magnitude and direction. For learned synthetic sets, however, the composed error decomposes exactly into this local discrepancy and an aggregate source residual. Under endpoint matching the residual exceeds the structural term by more than an order of magnitude, and under distribution matching the two terms partly cancel. Joint distillation also retains an accuracy advantage when both sets are distilled from the same dataset, where the local discrepancy is exactly zero. Training fidelity and downstream accuracy are therefore distinct requirements, neither established by evaluating each set on its own.