Visual grounding helps small language models learn object properties better
Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Computation and Language
Summary
This paper tests whether giving a small language model some visual clues about word meanings before training helps it learn better. The researchers gave image-based starting points for some words related to objects, like colors and shapes, then trained the model on a text dataset. They found that this visual seeding helped the model remember and predict properties of specific objects better, but it did not improve general grammar or abstract language skills. The paper raises the question of what kinds of tests would detect the benefits this kind of visual grounding provides.
What this means in practice
- •For natural language engineers: Improve small language models’ understanding of object properties by initially grounding some word embeddings in visual data.
- •For robotic system developers: Enhance robot language modules’ grasp of object attributes by integrating visual grounding during language model training.
Authors
Lisa Bylinina
Abstract
A language model normally begins training with random word embeddings: whatever 'banana' means must be learned from training corpora. I implement St. Augustine's picture of word learning, meaning by ostension, for a small masked language model (DeBERTa) trained on 10M words: before training, visually grounded tokens receive embeddings derived from the image regions they label; other tokens start random. Visual initialization leaves a measurable imprint that lasts until the end of training. At the same time, the effect remains invisible under most BabyLM benchmarks, which probe abstract grammatical knowledge: visual initialization does not affect performance there. The only zero-shot exception is object-property knowledge (COMPS, Misra et al. 2023), where seeding helps in every configuration. To follow up on this result, I build a corpus-tailored version of the Visual-Property Swap benchmark (Lin et al., 2026), which tests color, material, size, and shape knowledge, with per-item training frequency and seeded status. Here, vision-seeded models have a persistent, seed- replicated advantage, confined to the seeded words. As a causal test, I show that synthetic grounding of previously unseeded words transfers the advantage to exactly those words. Function words and abstract vocabulary also receive strong visual seeds and retain them throughout training, and the training objective draws on them: held-out mask-prediction loss falls for these words in every seed. However, no benchmark I run registers this. What evaluation would pick this up remains an open question.