The Geometry of Low-Resource Language Representations
2026-08-24 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied why large language models (LLMs) perform worse on languages with less training data by looking at the shapes of the internal language representations. They found that languages with less data have less distinct or 'degenerated' representations, especially in the model's final layers. To fix this, they added special penalties during further training to keep these representations clearer. Their tests on African languages showed this approach helps maintain better language representations and slightly improves performance on harder tasks. Overall, the authors show that the geometry of language data inside LLMs differs by resource level and can be improved with targeted training methods.
large language modelslow-resource languagesrepresentational geometryhidden representationscontinued pretrainingregularisationcosine similaritylanguage adaptationmodel degenerationAfrican languages
Authors
Francois Meyer, Jan Buys
Abstract
The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these disparities. In this paper, we characterise this gap through the lens of representational geometry. Comparing the geometric properties of hidden representations across 30 languages reveals that LLM geometry is systematically related to language data availability. The most consistent effect is in final layers, where low-resource languages exhibit representational degeneration. To counter this, we investigate the effectiveness of regularisation terms to penalise degeneration during continued pretraining (CPT). Experiments monolingually adapting 9 base LLMs to 10 African languages show that geometric regularisation successfully reduces representational degeneration during CPT. For larger models, cosine similarity-based regularisation marginally improves performance over vanilla CPT, with more consistent gains on the most challenging tasks. We establish that the representational geometry of low- and high-resource languages in LLMs is measurably distinct, and that targeted geometric intervention is a viable strategy for improving CPT for low-resource languages.