Medical visual encoders pretrained for improved report generation with language models
Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models
Artificial Intelligence
Summary
Multimodal language models combine images and text but often use visual encoders trained for general tasks, which don’t perfectly match how language models understand images. The authors developed MedMLIP, a way to train visual encoders specifically so they better communicate with language models by generating medical reports. They also introduced a technique called Local Relational Distillation to keep detailed image relationships intact. Their pretrained visual encoders showed better performance on medical image understanding tests and worked well with different language models.
What this means in practice
- •For medical imaging teams: Enhance automated medical image report generation systems by using pretrained visual encoders that better align with language model interfaces for more accurate interpretations.
- •For clinical ai developers: Develop clinical decision support tools that integrate improved visual encoders transferring seamlessly across different large language models for versatile multimodal diagnostics.
Authors
Tianyou Jiang
Abstract
Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consumed by autoregressive LLMs. We refer to this mismatch as the semantic-interface gap and introduce MedMLIP, a framework that pretrains the visual encoder through report generation with a frozen LLM, while employing Local Relational Distillation (LRD) to preserve relationships among visual patches to avoid visual collapse. We pretrain MedMLIP on IU-Xray and Open-PMC-300K and evaluate the resulting encoders on VQA-RAD and SLAKE. Only the ViT is transferred, while the guiding LLM and projector are replaced, allowing us to assess cross-LLM transferability. Our cross-LLM transfer experiments demonstrate the value of pretraining visual encoders for their autoregressive LLM interface while trying to preserve more fine-grained visual information. Code and the pretrained model are available at https://github.com/SkyCol/MedMLIP