Language models trained for specific tasks align better with brain regions
Cognitive Expert Language Models Better Align with the Corresponding Brain Systems
Computation and Language
Summary
Some parts of our brain are good at certain kinds of thinking, like understanding space or numbers. Usually, language models try to predict brain activity using one general approach for all brain areas. The paper shows that if you train language models to specialize in specific thinking types, they better match the activity in the brain parts that do those tasks. This was true even when tested with different models and brain data sets. It means that using specialized models gives a clearer view of how models relate to brain functions.
What this means in practice
- •For ai developers: Improve brain-inspired language models by building specialized models aligned to cognitive task areas for better regional brain prediction.
- •For neurotechnology engineers: Create brain-machine interfaces that decode cognitive domains more accurately by using domain-specific language model predictions.
Authors
Zhivar Sourati, Mengxuan Helen Wu, Nona Ghazizadeh, Jonas Kaplan, Morteza Dehghani, Samuel A. Nastase
Abstract
Large language models (LLMs) can predict human brain activity across a variety of brain regions during natural language comprehension. Typically, however, LLM-brain alignment is measured using one model for different regions of the brain, and then model performance is summarized across regions. This one-model-fits-all approach ignores the functional specialization of brain regions. In this study, we assess whether a model oriented toward a particular cognitive domain aligns better with the brain system dedicated to that domain. Through prompting and fine-tuning, we first build expert LLM variants for six domains: sensory, spatial, numerical, reasoning, social, and abstract processing. We then examine whether each expert best predicts activity in the brain region associated with the corresponding cognitive domain. Consistent with our hypotheses, each expert's representations align more closely with the brain system most associated with the matching domain than do other experts. This holds under both prompting and fine-tuning, across three base models and three fMRI datasets. In a series of control analyses, we show that this model-brain alignment is specific to cognitive domain interventions; non-cognitive and surface-level interventions do not result in comparable alignment. Specializing models shifts regional alignment while leaving aggregate prediction accuracy largely unchanged, suggesting that summarizing alignment across regions may obscure regional differences in performance for specific models.