Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
2026-06-08 • Computation and Language
Computation and Language
AI summaryⓘ
The authors developed a new method called Convex Gate (C-Gate) to connect speech signals to large language models (LLMs) without changing the LLM itself. Instead of converting speech to discrete tokens or unrestricted continuous forms, C-Gate represents sound using combinations of existing token embeddings, keeping the data compatible with the LLM. This approach improved both speech recognition and emotion detection tasks, showing that the way speech is represented over time in the embedding space matters more than fixed token identities. Their findings highlight the importance of geometric structure in speech-to-LLM integration and provide tools for further research.
Large Language ModelsSpeech RecognitionAcoustic Signal ProcessingEmbedding SpaceConvex CombinationAutoregressive DecodingParalinguistic InformationMultimodal IntegrationToken EmbeddingsFrozen Model
Authors
Ming-Hao Hsu, Yuxuan Hu, Shujie Liu, Jinyu Li, Yan Lu, Zhizheng Wu
Abstract
Large language models (LLMs) provide a powerful reasoning backbone for speech understanding, but integrating continuous acoustic signals into a frozen LLM remains challenging. Existing speech-to-LLM interfaces typically operate at two extremes: either enforcing near-discrete token alignment, which benefits transcription but loses paralinguistic information, or learning unconstrained continuous representations, which can drift away from the LLM's input space and degrade autoregressive decoding. In this work, we propose Convex Gate (C-Gate), a speech-to-LLM bridge that constrains all speech representations to lie within the LLM's input embedding manifold with an architectural convex-hull constraint. Concretely, each frame is represented as a convex combination of token embeddings, ensuring compatibility with the pretrained LLM while preserving continuous expressivity. Across automatic speech recognition (ASR) and emotion recognition, C-Gate achieves strong joint performance, improving LibriSpeech WER by up to 48.7% relative while matching or exceeding single-task emotion accuracy. Beyond performance, our analysis reveals a key insight: information is not carried by discrete token identities, but by time-resolved trajectories in the embedding space. Causal interventions confirm that both the trajectory structure and alignment to the pretrained embedding manifold are critical for performance. These results suggest that geometry, rather than token discreteness, is the fundamental design factor in speech-to-LLM interfaces, and provide a controlled regime for studying multimodal integration in frozen LLMs. We release the checkpoint, per-sample outputs, mechanism dumps, and intervention suite for replication.