Connecting Speech to Words through Images
2026-06-15 • Computation and Language
Computation and Language
AI summaryⓘ
The authors developed a method to learn how spoken words relate to their written forms without using text guides. They use pictures and spoken descriptions, first identifying important words related to images with captioning systems. Then, they find spoken sentences that include those words and align parts of the speech to the written words without direct text help. Their method works well for finding spoken words and beats some existing systems, showing promise especially for languages without text transcripts.
visually grounded learningimage captioningunsupervised word discoveryspoken word retrievalkeyword spottingspeech segmentationmultimodal learninglow-resource languagesspoken language processingalignment
Authors
Gabriel Pirlogeanu, Dan Oneata, Horia Cucu, Herman Kamper
Abstract
How can we learn the mapping between written words and their spoken counterparts in the absence of explicit textual supervision? We present a visually grounded method for building a vocabulary of spoken words using only images and their spoken descriptions. First, image captioning systems are used to build a vocabulary of written words representing salient visual concepts in the images. For each word, we then find utterances whose image captions contain that word. Then we use an unsupervised word discovery technique to align these utterances to locate instances of the target word. The result is spoken word segments that are linked to written words -- all accomplished without any text supervision. In spoken word retrieval and keyword spotting experiments, the proposed approach outperforms a strong neural baseline while being more interpretable. These results demonstrate the feasibility of the approach in English and motivate future work on low-resource languages without transcripts.