Visual grounding helps small language models learn object properties clearly
Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Computation and Language
Summary
Language models usually start learning words without any built-in meaning and figure it out from lots of text. The authors tried giving a small language model some initial help by linking certain word meanings to images before training. They found this visual help makes the model remember object properties like color and shape better, but it does not improve understanding of grammar or abstract words in usual tests. The visual advantage sticks around only for words directly linked to the images. The authors also noticed some effects that current tests do not capture, and they suggest new ways are needed to measure these.
language modelword embeddingsostensive definitionvisual groundingmasked language modeltraining corpusobject propertiesevaluation benchmarksabstract vocabularyzero-shot learning
Authors
Lisa Bylinina
Abstract
A language model normally begins training with random word embeddings: whatever 'banana' means must be learned from training corpora. I implement St. Augustine's picture of word learning, meaning by ostension, for a small masked language model (DeBERTa) trained on 10M words: before training, visually grounded tokens receive embeddings derived from the image regions they label; other tokens start random. Visual initialization leaves a measurable imprint that lasts until the end of training. At the same time, the effect remains invisible under most BabyLM benchmarks, which probe abstract grammatical knowledge: visual initialization does not affect performance there. The only zero-shot exception is object-property knowledge (COMPS, Misra et al. 2023), where seeding helps in every configuration. To follow up on this result, I build a corpus-tailored version of the Visual-Property Swap benchmark (Lin et al., 2026), which tests color, material, size, and shape knowledge, with per-item training frequency and seeded status. Here, vision-seeded models have a persistent, seed- replicated advantage, confined to the seeded words. As a causal test, I show that synthetic grounding of previously unseeded words transfers the advantage to exactly those words. Function words and abstract vocabulary also receive strong visual seeds and retain them throughout training, and the training objective draws on them: held-out mask-prediction loss falls for these words in every seed. However, no benchmark I run registers this. What evaluation would pick this up remains an open question.