Norwegian further-trained model does not give more outdated legal values
Does a Norwegian further-trained model give more outdated legal values than its base model? A preregistered paired comparison of Borealis 2 and Gemma 4
Computation and Language
Summary
This study looked at two computer language models that answer questions about Norwegian laws to see if one gave older, outdated legal numbers more often. One model had been further trained by the National Library of Norway, while the other was its original base model. The researchers found that neither model gave outdated answers more often than the other in any meaningful way. This means further training the model did not make it less up-to-date in this specific legal domain.
What this means in practice
- •For legal technology developers: Assess the impact of further training on accuracy of language models answering legal numeric questions using the norregel benchmark.
- •For government data teams: Use these findings to choose between base or further-trained language models when ensuring current legal values in automated systems.
Authors
Eirik Botten Nicolaysen
Abstract
We compare how often two released language models give outdated numeric values from Norwegian statutes and regulations: NbAiLab/borealis2-26b-a4b-preview (Borealis 2), which the National Library of Norway further trained from google/gemma-4-26B-A4B-it (Gemma 4), and Gemma 4 itself. The study was preregistered (doi:10.5281/zenodo.23858965) and uses the closed norregel benchmark, scored deterministically against a value history. On the 411 of 503 primary items where both models gave an interpretable answer, Borealis 2 alone was outdated on 10 items and Gemma 4 alone on 11 (d = -0.2 percentage points, 95% cluster bootstrap interval [-2.7, +2.5], exact one-sided McNemar p = 0.668). By the preregistered rule this is no effect of interest: the whole interval lies within the smallest effect of interest of +/-10 percentage points. Among the secondary measures, only the share of answers giving a value that took effect after the question date differs after Holm correction; Gemma 4 gave such answers more often. An exploratory comparison of two smaller Borealis models had no discordant pairs and is not informative. The design compares released models and does not isolate the effect of any part of their training.