VGL-stGen connects lung tissue images with gene activity patterns
VGL-stGen: Learning a shared multimodal model for lung histology and spatial transcriptomics
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding how lung tissue looks and how genes are active inside is important for disease research. The authors created VGL-stGen, a computer model that learns from both tissue pictures and gene information at the same time. This lets the model guess cell types, predict gene activity, and fill in missing gene data based on images and gene clues. It helps link what the lung tissue looks like under a microscope to the underlying molecular activities.
What this means in practice
- •For clinical pathology labs: Use combined tissue image and gene data models to improve diagnostic classification of lung diseases.
- •For biotechnology companies: Develop tools that integrate molecular and histological data to guide drug target validation in lung research.$Commercial implications: Enables creation of products linking tissue morphology to gene activity for lung disease drug development.
Authors
Haenara Shin, Dongjoo Lee, Sungwoo Bae, Kwon Joong Na, Daeseung Lee, Hongyoon Choi
Abstract
VGL-stGen is a multimodal large language model adapted from a pretrained medical vision-language model using paired lung H&E and spatial transcriptomics with single-cell references. Training integrates image-expression and expression-cell-type pairs, and gene-conditioned pseudo-labels transfer transcriptomic cell-type supervision to histology images. One deployed checkpoint supports thirteen evaluated tasks spanning tissue and cell-type classification, gene prediction and conditional expression completion from combinations of histology, cell identity and observed gene expression. Task-specific decoding produces categorical labels, gene lists and structured expression levels. For conditional completion, fixing queried gene identities and predicting their levels autoregressively improves structured prediction. Controlled adaptation shows that restricting supervision to requested expression-level tokens can further improve image-conditioned completion. Image-permutation experiments indicate that correctly paired histology adds information beyond observed expression. External classification relies partly on computational references with shared annotation lineage, and some H&E sources were exposed during cancer training or model development. Within these conditions, VGL-stGen demonstrates how a medical vision-language backbone can learn from spatial transcriptomics and histology to connect tissue morphology and molecular states through shared generative modeling.