Convex combinations of text embeddings improve speech and audio models
ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence
Computation and LanguageSound
Summary
Connecting speech sounds to text is hard because models usually transform sounds into new forms they must learn to understand. The authors of this paper made a new way to turn sounds into mixtures of existing text meanings that a model already knows. This helps the model better use what it learned from text while understanding spoken questions and reasoning tasks. Their approach, ConvexLLM, works well on many speech and audio tasks, especially when dealing with spoken knowledge.
What this means in practice
- •For voice assistant developers: Improve voice assistants’ ability to understand complex spoken questions by using convex combinations of text embeddings.
- •For audio content indexing teams: Enhance indexing of spoken knowledge content by leveraging audio models that better preserve text understanding.
Authors
Ming-Hao Hsu, Shujie Liu, Yuxuan Hu, Yuxiang Wang, Yingda Shen, Jinyu Li, Zhizheng Wu
Abstract
Speech and audio language models connect continuous acoustic representations to decoders pretrained primarily on text data. Most such models use MLP or Q-Former interfaces to map acoustic features into the decoder's input space, requiring the decoder to learn how to interpret the resulting representations. We investigate whether grounding these representations in the decoder's existing text embeddings can help it use acoustic information while better preserving its pretrained text capabilities. We present ConvexLLM, a general-purpose speech and audio language model whose C-Gate interface represents audio as continuous convex combinations of frozen text embeddings. ConvexLLM achieves an eleven-task VoiceBench aggregate of 75.36, the highest among the open-weight models we compare, and an MMAU Test-mini accuracy of 62.50%. Its largest margins over the baselines are on spoken knowledge and reasoning tasks, while the strongest baselines are more accurate on audio understanding and transcription. In a comparison with a 2B-parameter decoder, C-Gate achieves higher aggregate spoken-language performance than MLP and Q-Former interfaces, with text-task performance close to MLP and task-dependent tradeoffs in transcription and audio understanding. These results support convex combinations of text embeddings as a viable interface for general-purpose speech and audio language models.