Language model mimics brain neurons for efficient learning from small data
URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining
Computation and Language
Summary
Most language models learn from huge amounts of text data, unlike the human brain which learns language from much less data. The authors created URCHIN, a language model designed to work like the brain’s neurons, using spiking neurons organized in a biologically realistic way. URCHIN uses a small network mimicking excitatory and inhibitory brain neurons to understand and generate language efficiently with limited data. It runs efficiently on GPUs for training and on CPUs or special hardware for practical use, matching benchmarks on child-scale language tasks.
What this means in practice
- •For embedded system engineers: Run efficient language models on low-power edge devices using spiking neuron architectures without converting weights between training and deployment.
- •For gpu software developers: Train biologically inspired spiking language models efficiently using parallelizable state-space algorithms on standard GPU hardware.
Authors
Po-Han Chiang
Abstract
The BabyLM challenge measures how much language a model can learn from developmentally-plausible, child-scale data rather than internet-scale corpora, yet prior language models forgo the biological constraints of the neural circuitry that acquires human language: spiking neurons separated into excitatory and inhibitory populations wired by a recurrent lateral connectome. This paper presents URCHIN (Unified Recurrent Connectome with Horizontal Integrate-and-fire Neurons), which applies the Parallelized Hierarchical Connectome Spiking State-space Model (PHCSSM) to language modeling: leaky integrate-and-fire neurons coupled by a Dale's-law lateral connectome resolve each token through a multi-transmission loop that recirculates activity to a fixed point. The instantiation is deliberately minimal: a single horizontal layer of 128 neurons, no attention, and 4.23M parameters. Two implementations share one set of weights and produce identical benchmark scores, so URCHIN is trained once and deployed either way with no conversion step: a parallel state-space model (SSM) scan that is GPU-efficient for training, or an event-driven recurrent spiking neural network (RSNN) with constant-cost inference for CPU or neuromorphic edge deployment. Across all three BabyLM tracks (Strict-100M, Strict-Small, and Multilingual), URCHIN offers a biologically plausible, efficient, and directly deployable reference point.