A General-Purpose VLM Can Teach an Astronomy Foundation Model to Better Recognize Galaxy Morphology
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors show that visual language models (VLMs), which combine images and text understanding, can help improve galaxy shape classification even when few human-labeled examples are available. They created a new test set focused on galaxy images from different surveys and found that VLMs know some useful galaxy features but aren't yet accurate enough to replace human labels. By using a VLM as a 'teacher' to guide an astronomy model called Zoobot, they improved galaxy classification across different datasets with limited labels. Their approach aims to make it easier to adapt to new large sky surveys without needing lots of manual labeling.
Visual Language Models (VLM)Galaxy MorphologyWeak SupervisionVisual Question Answering (VQA)ZoobotGalaxy ZooSurvey AdaptationLarge-scale Sky SurveysLegacy Survey of Space and Time (LSST)Nancy Grace Roman Space Telescope
Authors
Dichang Zhang, Jiaqi Deng, Yixuan Shao, Yuanpeng Liu, Jiali Cui, Zhiqiang Lao, Heather Yu, Liang Peng, Simon Birrer, Dimitris Samaras
Abstract
Existing astronomy foundation models provide strong galaxy representations, but adapting them to new survey conditions and survey-specific morphology recognition tasks still requires substantial human supervision. We show that VLM-based VQA systems contain meaningful visual-semantic priors that can serve as weak supervision for downstream morphology classifiers and improve morphology classification under limited human-label budgets. We first introduce a survey-oriented VQA benchmark spanning two representative imaging regimes and evaluate state-of-the-art VLMs on galaxy morphology questions. The results show that these models capture useful morphology signals and informative uncertainty, but are not sufficiently reliable to replace human annotators. Motivated by this finding, we use a general-purpose VLM as a morphology teacher for Zoobot, an astronomy foundation model pretrained on large-scale Galaxy Zoo annotations. Across two survey domains and multiple annotation budgets, the VLM teacher consistently improves Zoobot's downstream morphology classification. These results demonstrate that a general-purpose VLM provides knowledge complementary to an astronomy foundation model and can teach it to better recognize galaxy morphology under limited human supervision. The resulting pipeline is designed for label-efficient adaptation to forthcoming large-scale surveys, including the Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) and the Nancy Grace Roman Space Telescope. The benchmark and code are publicly available at https://github.com/fw-ic/VLM-morphology-teacher.