Chameleon adapts number formats to improve diffusion model efficiency
Chameleon: Dynamic Format Adapter for Efficient Diffusion
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Diffusion models, used for tasks like image generation, often need to run on devices with limited memory, which requires compressing their data without losing quality. Existing methods fix the type of number format in advance and only adjust simple parameters, but this overlooks how different parts of the model benefit from different formats. The authors propose Chameleon, which dynamically chooses the best number format per part of the model based on data statistics, improving efficiency and output quality. Their tests showed Chameleon consistently delivers better results across several popular diffusion models.
What this means in practice
- •For machine learning engineers: Deploy diffusion-based image generation models more efficiently on memory-limited hardware by dynamically adapting number formats per model component.
- •For embedded system developers: Improve performance and output quality of AI models used in devices with strict memory and compute constraints by applying adaptive quantization strategies.
Authors
Arnab Sanyal, Sandeep Chinchali
Abstract
Post-training quantization (PTQ) is the standard way to run modern diffusion models on memory-constrained accelerators, yet every existing diffusion PTQ scheme fixes the $\mathit{number\ format}$ in advance and only tunes the scale, zero point, or per-layer bit-width. At a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs across weight channels, across layers, and along the diffusion timestep, where activation distributions slide from heavy-tailed and noise-dominated to tightly clustered and structured. We propose Chameleon, a PTQ framework that holds the bit-width fixed and treats the format itself as a discrete variable, chosen per weight channel and per (layer, timestep bucket) activation tensor. Activation formats come from {INT8, FP8 E4M3, FP8 E5M2, MXFP8, MXINT8}, selected ahead of time from two cheap statistics (empirical kurtosis and the closed-form diffusion SNR) and stored in a lookup table; weight formats come from {INT8, MXINT8} at 8 bits or {INT4, NF4, FP4 E2M1, MXINT4, MXFP4} at 4 bits, selected offline by reconstruction error. An architectural fork adapts the same selection layer to multi-step UNets, single-step distilled models, and Diffusion Transformers. Across SDXL, SDXL-Turbo, and PixArt-$α$ on COCO-2014, Chameleon achieves the best FID in all six backbone $\times$ bit-width settings, with CLIP within 0.24 of the FP16 reference and the best of all quantized methods at $W_{4}A_{8}$.