Summary
Understanding climate-related text is tricky because it comes from many sources like science papers, online content, and reports created by computers. The authors worked on making a language model better at this by training it on different types of climate texts. They found that combining academic climate writings with web data and synthetic documents helped the model perform better on climate tasks. They also tried training separate models on each source and then merging them, which worked better than mixing all the texts at once. Their work shows the best way to train computer models for understanding climate information and they shared their models for others to use.
Natural language processingContinued pretrainingLanguage modelsClimate domainDomain adaptationCorpus compositionParameter-space mergingEncoder modelsF1 score
Authors
Yongan Yu, Shantam Raj, Jingwei Ni, Ario Saeid Vaghefi, Dominik Stammbach, Markus Leippold
Abstract
Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored. We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. We systematically compare joint continued pretraining on corpus mixtures with parameter-space merging of independently specialized checkpoints. Across nine climate NLP benchmarks, our best model achieves 76.3 average F_1, improving significantly over a vanilla ModernBERT baseline by 2.8 points. Within the climate NLP setting, the results show that academic climate corpora provide the strongest adaptation signal among the evaluated sources, while parameter-space merging improves over joint multi-source training and better preserves complementary information from heterogeneous climate corpora. We release all Climate-ModernBERT variants and training checkpoints to support future research in climate NLP and domain-adaptive pretraining.