Climate adapted language models improve understanding of climate texts
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
Computation and LanguageArtificial Intelligence
Summary
Understanding climate-related texts is tricky because they come from different sources like scientific papers, websites, and simulated reports. The authors created Climate-ModernBERT, a version of a language model trained specifically on diverse climate texts to handle this variety better. They tested different ways to combine these texts during training and found that merging specialized models preserves useful information best. This new model performs better on climate-related language tasks than the original version.
What this means in practice
- •For climate data engineers: Build improved text analysis tools that better interpret climate science and policy documents using domain-adapted language models.
- •For environmental policy developers: Use enhanced language models to automate extraction of policy-relevant information from diverse climate-related texts, supporting policy formulation.
Authors
Yongan Yu, Shantam Raj, Jingwei Ni, Ario Saeid Vaghefi, Dominik Stammbach, Markus Leippold
Abstract
Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored. We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. We systematically compare joint continued pretraining on corpus mixtures with parameter-space merging of independently specialized checkpoints. Across nine climate NLP benchmarks, our best model achieves 76.3 average F_1, improving significantly over a vanilla ModernBERT baseline by 2.8 points. Within the climate NLP setting, the results show that academic climate corpora provide the strongest adaptation signal among the evaluated sources, while parameter-space merging improves over joint multi-source training and better preserves complementary information from heterogeneous climate corpora. We release all Climate-ModernBERT variants and training checkpoints to support future research in climate NLP and domain-adaptive pretraining.