MedVTok unifies medical images into one token system for AI
MedVTok: A General-Purpose Medical Visual Tokenizer
Computer Vision and Pattern Recognition
Summary
Medical AI systems often struggle because images from different types of scans and formats use different ways to understand them. The authors created MedVTok, a tool that turns many kinds of medical images—like 2D pictures and 3D scans—into a common language that AI can understand better. They focused on keeping the anatomical details consistent across slices and combining expert knowledge about different scan types. This helps MedVTok work well on many tasks like identifying diseases, segmenting images, or answering questions about medical images.
What this means in practice
- •For medical ai developers: Build AI systems that handle multiple medical image types and dimensions using one shared encoding method.
- •For radiology software engineers: Create tools that improve image classification, segmentation, and report generation by using a unified image tokenizer.
Authors
Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, Yujie Wei, Shaohao Rui, Chaoyang Zhang, Wenjie Li, Wenhao Tang, Cheng Tang, Wei Li, Shujian Gao, Yujie Zhang, Ying Chen, Zilong Li, Zongyuan Ge, Diping Song, Jin Ye, Ming Hu, Junjun He, Hongming Shan
Abstract
Multimodal medical AI requires a common visual interface for encoding heterogeneous imaging evidence for patient condition modeling. However, existing medical visual tokenizers are often tied to a specific dimension and imaging modality, forcing multimodal systems to rely on fragmented and poorly reusable representations. We introduce MedVTok, a general-purpose visual tokenizer that maps 2D images and 3D volumes from diverse imaging modalities into a unified token space. Building such a tokenizer poses two key challenges: preserving anatomical structure across dimensions and capturing clinical semantics across modalities. To this end, MedVTok introduces (i) slice differential regularization, which explicitly models inter-slice anatomical coherence missing in previous per-pixel/per-voxel supervision, and (ii) multi-expert representation alignment, which integrates knowledge from multiple medical expert encoders while retaining modality-specific complementary knowledge. Trained on 28M images and 70K volumes across 9 imaging modalities, MedVTok supports classification, retrieval, segmentation, synthesis, visual question answering, and medical report generation through a single tokenizer. Across 24 benchmarks, MedVTok achieves state-of-the-art performance, demonstrating a scalable token interface that unifies medical image dimension, modality, and task usage.