AI summaryⓘ
The authors studied how to make large language models better at understanding and working with Swedish journalism by training them further on a big set of news articles. They created new tests focused on specific journalism tasks to check if this extra training helped. Their results showed that additional training improved the models' ability to generate text and recall facts, but not their skill at tasks requiring distinguishing between options. They also found that using experience replay during training helped prevent the model from forgetting old information, and that some special fine-tuning methods gave extra benefits. Finally, the authors highlighted the need for tailored tests to properly measure improvements in specific fields.
large language modelscontinued pre-trainingSwedish journalismexperience replaygeneration qualityfactual knowledgediscriminative tasksparameter-efficient fine-tuninginstruction followingdomain-specific benchmark
Authors
Lukas Borggren, Jenny Kunz, Marco Kuhlmann
Abstract
Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large language models to Swedish journalism, using a high-quality dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the models' generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish benchmark largely fails to capture the models' in-domain performance gains.