Medical image tokenizers show linked reconstruction and generation quality
What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Medical images are compressed into smaller pieces called tokens before being generated or analyzed, but tools made for everyday photos are often used without testing if they work well for medical images. The authors tested many different tokenizers on various medical datasets and found that how well a tokenizer reconstructs images is closely connected to how well it can generate new ones. They also found most tokenizers use their token sets fully but don’t explore many possible combinations, and that preserving important image details for diagnosis is possible even with compression.
What this means in practice
- •For medical imaging developers: Improve medical image generation pipelines by choosing tokenizers that best preserve reconstruction and classification quality for smaller, less variable datasets.
- •For digital pathology teams: Select discrete quantization methods for image tokenization that retain key diagnostic features in compressed medical images used for automated analysis.
Authors
Niklas Bubeck, Yundi Zhang, Vasiliki Sideri-Lampretsa, Julian McGinnis, Jiancheng Yang, Daniel Rueckert, Jiazhen Pan
Abstract
Latent diffusion models now dominate medical image generation, and every such pipeline rests on a \emph{tokenizer} that compresses images into the latent codes for image generation to operate on. Thereby, the tokenizer choice bounds every downstream task from reconstruction fidelity and generation quality to the representations available for downstream analysis. Yet, medical imaging pipelines routinely utilize tokenizers from natural imaging on the hypothesis that their behavior carries over. However, this is an assumption never tested in the medical imaging regime, where datasets are orders of magnitude smaller and images exhibit far lower inter-sample variance. We present a systematic evaluation of medical image tokenizers evaluating thirty configurations across ten model families on twelve datasets at three compression factors, spanning reconstruction, generation, latent geometry, downstream classification, and memorization. We find that (1) performance on image reconstruction and generation strongly correlate, unlike prior reports on natural images; (2) modern tokenizers use nearly all of their codebook entries, but still leave most of the latent space unused; (3) training-set memorization is mild and is further suppressed by stronger latent space compression; and (4) discrete quantization can largely preserve downstream classification, with lookup-free schemes being the main exception.