Latent space geometry improves 3d medical image synthesis accuracy
Latent Space Is Not Flat: Rethinking Latent Structure for 3D Medical Image Synthesis
Computer Vision and Pattern Recognition
Summary
Generating 3D medical images using computers often involves creating smaller versions called latent spaces. The authors found that these latent spaces aren't simple flat areas but have complex shapes that affect how images are created. They discovered that variations in latent space mostly happen in a small set of directions that capture important structures, while other parts capture noise or small details. To use this insight, the authors designed a new method called Latent Structure Flow that treats these two parts differently during image generation, improving results for tasks like combining different image types or filling in missing tumor regions.
What this means in practice
- •For medical imaging teams: Improve accuracy when synthesizing 3D medical images across different imaging techniques and filling in missing tumor regions.
- •For radiology software developers: Enhance 3D image reconstruction tools by using latent space structure modeling to reduce decoding errors in generated images.
Authors
Haowen Xue, Hao Chen, Hexuan Hu, Qian Huang, Yi Han, Qing Meng, Zaipeng Xie, Chao Li, Haoli Xu
Abstract
Latent generative models make 3D medical image synthesis computationally practical by generating in a compressed space. However, we show that the common flat Euclidean assumption induced by $\ell_2$ objectives is imprecise: latent-space geometry is so strongly anisotropic that equal-magnitude errors can produce drastically different decoded distortions. We further find that this anisotropy has a clear feature: sensitive variation concentrates in a low-rank subspace. The dominant low-rank components capture the overall structure, encoding long-range, spatially coordinated variation while remaining resistant to local noise. Its orthogonal residual, in contrast, mainly captures local and image-specific variation. Motivated by this asymmetry, we introduce Latent Structure Flow (LSF). At each block, LSF decomposes the latent state into structure and residual, models structural changes with global context, and predicts residual variation locally while preserving a direct path for the input structure. LSF changes only the generator, leaving the frozen codec and pointwise training objective unchanged. Across cross-modality synthesis and tumor inpainting tasks, LSF outperforms all compared baselines on both global and tumor-specific metrics, demonstrating the benefit of explicitly modeling latent-space structure for 3D medical image synthesis.