Earth system data and language are aligned with EarthAlign and MeteoLIP
EarthAlign: Learning Language-Aligned Representations of Earth System Data
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Understanding Earth's weather and climate is important for many decisions, but it is hard to connect detailed weather data with natural language descriptions. The authors created EarthAlign, a large dataset linking weather data with expert-written captions about events and normal conditions from 2000 to 2024. They also developed MeteoLIP, a computer model that learns to connect weather data with language. Their tests show this model works better than earlier methods for tasks like matching weather with text and estimating conditions from language.
What this means in practice
- •For weather data engineers: Integrate EarthAlign and MeteoLIP to improve matching of natural language weather reports with meteorological data for forecasting and analysis pipelines.
- •For disaster response teams: Use aligned weather-language models to quickly retrieve relevant meteorological conditions from textual event descriptions to support disaster assessment.
Authors
Fadillah Adamsyah Maani, Numan Saeed, Jiaqi Ma, Dat Tien Nguyen, Youssef Wehbe, Duncan Watson-Parris, Salman Khan
Abstract
Interpreting Earth system conditions is central to understanding environmental risks and informing decisions across weather- and climate-sensitive sectors. Existing multimodal weather systems primarily target task-specific prediction, generation, or question answering, while general-purpose alignment between Earth system data and language remains underexplored due to the limited availability of diverse language supervision grounded in meteorological conditions. To fill this gap, we introduce EarthAlign, a large-scale dataset with broad geographic coverage spanning 2000-2024 that combines 10 expert-curated natural event and disaster data sources with climatology-derived normal-weather samples. Each sample includes temporal and geospatial information and four grounded caption styles aligned with 63-variable ERA5 meteorological data. Building on EarthAlign, we introduce MeteoLIP, a dual-encoder framework for learning aligned meteorology-language representations. MeteoLIP incorporates MetTok, which uses a geolocation- and time-conditioned query to aggregate variable-specific patch embeddings into a compact sequence of meteorology tokens. To comprehensively assess Earth system multimodal representations, we introduce three downstream tasks: cross-modal retrieval, weather-condition classification, and physical variable estimation. Our experiments show that MeteoLIP improves upon the corresponding CLIP and SigLIP baselines across these tasks. Further analyses highlight the importance of meteorological input, temporal and geospatial conditioning, and multi-style supervision for meteorology-language alignment. Thus, EarthAlign and MeteoLIP provide a foundation for learning and evaluating language-aligned representations of Earth system data. Our code, dataset, and weights are publicly available at https://github.com/mbzuai-oryx/EarthAlign.