AI summaryⓘ
The authors created a new method to identify cell types from regular microscope images of tissue stained with H&E by using detailed molecular data as a guide instead of relying on slow and costly manual labeling. They combined spatial transcriptomics data, which maps gene activity to specific cells, with corresponding H&E images across many organs and trained a model called CytoFormer on this large dataset. This model can accurately predict cell types in new tissue samples and works better than existing models even for unseen organs and cell types. Additionally, CytoFormer needs fewer labeled examples to learn, making it a useful and efficient tool for analyzing single cells in common tissue images.
Hematoxylin and eosin (H&E) stainingSpatial transcriptomicsCell type classificationCytoFormerFoundation modelSingle-cell analysisBiopsy tissueMachine learningLabel efficiencyHistology
Authors
Jialu Yao, Songhao Li, Alina Yu, Zhi Huang
Abstract
Identifying cell types directly from routine haematoxylin and eosin (H&E) histology would enable single-cell analysis at scale, but training such models has relied on manual pathologist annotations, which are slow, expensive and unreliable for many cell types. We instead supervise morphology with molecules. Imaging-based spatial transcriptomics profiles individual cells in situ on a section that can afterwards be stained with H&E, so that molecular identity and morphology are observed for the same physical cell. We assembled 81 such paired Xenium sections spanning 16 organs, derived per-cell labels by clustering, marker-gene annotation, organ-wise human review and quality control, and mapped them onto the cell types commonly reported in each organ. This yielded 15.4 million cells, each with a paired H&E image patch and one of 23 cell types, on which we trained CytoFormer, a cell foundation model with a multi-task, per-organ classification head. On spatially held-out tissue CytoFormer reached an accuracy of 0.85 and a macro-F1 of 0.78 across all 16 organs, and its predictions reproduced the tissue architecture of an entire held-out section. The representation also transfers: with the encoder frozen, a linear head on CytoFormer features performed better than six pathology foundation models on four expert-annotated benchmarks, including on organs and cell types that were not part of pretraining. Finally, in an interactive active-learning setting, CytoFormer's embeddings are markedly more label-efficient than existing pathology foundation models, detecting normal epithelium amid look-alike tumour with an F1 of 0.82 from only a few annotations and leading the strongest baseline by 0.13 in F1. CytoFormer turns paired H&E and spatial transcriptomics into a reusable, label-efficient representation for cell-level analysis of routine histology.