Invertible adapters improve latent space alignment for medical image generation
Semantic Latent Alignment for Flow Matching with Invertible Bi-Lipschitz Adapters
Computer Vision and Pattern RecognitionMachine Learning
Summary
It can be hard to adjust a computer’s understanding of compressed images to do new tasks without retraining everything, which is costly and tricky. The researchers found a way to keep the image encoder and decoder fixed but change the coordinates between them using a special invertible adapter that preserves structure and semantic meaning. This approach improves image features and stability in medical images while avoiding distortions that can cause problems. Their method is a simpler, more efficient way to adapt pretrained models for new image generation tasks without retraining the whole system.
What this means in practice
- •For medical image analysis teams: Enhance semantic consistency and stability when generating or editing medical images using pretrained models with constrained latent space adaptation.
- •For machine learning engineers: Adapt pretrained generative models to new tasks efficiently by applying bi-Lipschitz invertible adapters without expensive retraining or loss of reconstruction quality.
Authors
Felix Nützel, Mischa Dombrowski, Luca Hagen, Bernhard Kainz
Abstract
Pretrained latent spaces trade compression, reconstruction, and the learnability of the coordinates a downstream generator has to work in, which complicates adapting these spaces to different domains or tasks: continued training is expensive, and without a new alignment objective it often fails to improve downstream generation. We instead leave the encoder and decoder fixed and change only the coordinates between them, using an invertible bi-Lipschitz adapter aligned offline to a semantic teacher and frozen during class-conditional flow matching. Invertibility preserves compatibility with the pretrained autoencoder, while the bi-Lipschitz constraint limits how far alignment can distort its latent geometry. This provides a simple way to improve semantic organization while retaining the compression and reconstruction properties of the original latent space. On two medical imaging datasets, constrained latent adaptation improves feature metrics, seed stability and counterfactual edit strength, while remaining competitive on downstream classification. In contrast, unconstrained adaptation can become highly anisotropic and sensitive despite remaining invertible, highlighting the importance of controlling geometric distortion. Our results suggest that carefully constrained post-hoc adaptation offers a parameter-efficient and tuning-free alternative to jointly modifying the representation during generative training.