Relational attention improves language models with less data
Relational Attention for Data-Efficient Language Modeling
Computation and LanguageMachine Learning
Summary
Language models usually learn from a lot of text, but this work shows a way to make them learn better with less data. The researchers combined two types of attention in a Transformer model to better separate word meanings from sentence structure. They also added a training step to help the model remember and predict what's coming next more efficiently. Their method ranked well among many entries using a limited amount of training data, especially on language tasks requiring understanding relationships.
What this means in practice
- •For natural language processing teams: Build language models that generalize well with limited text data by integrating relational attention and Next-Latent Prediction training.
- •For conversational ai developers: Develop chatbots with improved understanding of relational language structure using data-efficient Transformer architectures.
Authors
Adrian Brasoveanu, Ece Takmaz, Jakub Dotlačil
Abstract
We present Relational BabyLM, a system submission to the BabyLM 2026 challenge that combines two cognitively motivated inductive biases in a single decoder-only Transformer. Architecturally, we replace standard self-attention with a Dual Attention Transformer (DAT), which separates the routing of object-level ("sensory") lexical features from structural/relational information (Altabaa and Lafferty, 2025; Altabaa et al., 2024; Webb et al., 2024; Kerg et al., 2022; Webb et al., 2021). Relational attention (RA) disentangled from self-attention greatly increases data efficiency and out-of-training-sample generalization on purely relational tasks, but language modeling requires object-level and relational information to be integrated as well as disentangled, and RA-based LMs have remained largely unexplored. BabyLM's data-constrained training and comprehensive evaluation is an ideal testing ground for whether that data efficiency transfers. As a training intervention, we add a Next-Latent Prediction (NextLat; Teoh et al. 2026) objective that encourages hidden states to compress history incrementally into a dense belief state. Architecture is the dominant factor for structural linguistic generalization; the objective is secondary but still significant. DAT's three relational attention types (full RA vs. the simpler RCA and DisRCA variants) are largely interchangeable at 10M words; full RA pulls ahead at 100M. We also introduce a novel symbol-retrieval mechanism (RoPE-based, as opposed to learned, relative symbols) that matches learned symbol libraries while adding no parameters. On the strict (100M-word) track, our best model ranks 6th of 55 overall and 3rd of 55 on the leaderboard's NLP-task subset at the time of writing; our two strongest models outperform the GPT-2 baseline on most benchmarks, with one attaining the highest EWoK score among strict-track entries.