Small language models learn better from child-specific speech data
Specialization, Diversity, and Scaling in Child-Scale Language Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Language models usually learn from huge amounts of text, but children hear much less language in daily life. This paper studies how small language models learn from the kind of language children actually hear. The authors found that models trained on one child’s speech predict that child’s language better than others’, showing specialization. Using data from several children helps models predict pooled language better but does not improve grammar much. The results highlight that learning depends on both the amount and the source of data, as well as how well training matches the type of language to predict.
What this means in practice
- •For speech recognition developers: Improve speech models for child-directed language by training on child-specific datasets to better match real child language environments.
- •For human-computer interaction designers: Design conversational agents that adapt to children’s unique language patterns by using small, specialized language models trained on child speech data.
Authors
Steven Y. Feng, Michael C. Frank
Abstract
Modern language models are trained on far more language than children experience, but we know much less about learning from small, naturalistic datasets. We train small GPT-2 models on transcripts of speech with children from dense home recordings (BabyView) and on synthetic child-directed dialogue (TinyDialogues), varying the amount, source, and diversity of training data. Models trained on one child's language environment predict that environment substantially better than other children's environments. Much of this specialization is lexical and distributional: a simple 5-gram model reproduces the cross-child pattern (r=.91). Training on data from multiple children clearly improves prediction of pooled held-out language relative to training on a single child's input without a similarly clear improvement on grammatical benchmarks. Across data sizes, naturalistic and synthetic training inputs also produce different grammatical and lexical-semantic patterns, with important differences between grammatical phenomena. Together, these results show that small-data language learning depends not only on how much language a model sees, but on which environments it comes from and how well that input matches what the model is asked to predict.