Multimodal models struggle with astronomy image question answering tasks
Star Light, Star Bright: Can Multimodal Models Parse an Astronomy Visual Question Answering Dataset Right?
Computer Vision and Pattern RecognitionDigital LibrariesInformation Retrieval
Summary
Scientists tested if existing multimodal AI models can understand and answer questions about images of the night sky. They created a special dataset of real, fake, and altered astronomical pictures with thousands of questions. The AI models often gave incorrect answers, showing they have trouble with domain-specific science images. The authors also compared AI errors to human errors on historical images and found similarities in certain measurements.
What this means in practice
- •For astronomy software developers: Develop tools that assess and improve AI models specifically for interpreting astronomical images with domain-focused question answering.
- •For scientific publishing teams: Create better automated quality control systems for digitized astronomy figures by understanding AI model limits on degraded images.
Authors
J. P. Naiman
Abstract
Figures are essential for scientific communication, and the field of Visual Question Answering (VQA) aims to test the figure understanding capabilities of large multimodal models (LMMs). Here, we investigate the accuracy of several low-cost LMMs on domain-specific VQA tasks for astronomical images of the sky. As instrument for this task we build a domain-specific VQA dataset comprising 2001 partially-synthetic figures divided evenly between contour plots, real telescope images drawn from NASA's SkyView, and "fake" sky images which share the axes and angular scales of the real images but draw their colors from a gaussian mixture model, alongside 47357 questions. Each figure is released as pristine and with realistic page-aging degradations mimicking the historical astronomical corpus. Testing three low-cost proprietary LMMs, we find models frequently fail to answer domain-specific questions at high accuracy. An initial comparison with 104 human-verified, astrometrically tagged figures from historical journal publications shows similar median errors in angular extent. All code is available on GitHub: https://github.com/jnaiman/SkyImagesWASP2026 and dataset is available on HuggingFace: https://huggingface.co/datasets/ReadingTimeMachine/astro_sky_image_vqa.