Brain audio and text signals aligned for better speech decoding
BAT-CLIP: Trimodal Alignment of Brain, Audio and Text
SoundArtificial IntelligenceMachine Learning
Summary
Decoding speech from brain signals usually links brain activity to either sound or text, but not both at the same time. The authors developed BAT-CLIP to connect brain signals with both audio and text together, which better reflects how the brain processes speech. Their method produces stronger and more informative brain signal representations than earlier two-way methods. They also show that using models trained without explicit labels helps this alignment work better.
What this means in practice
- •For brain-computer interface teams: Improve decoding of natural speech from brain signals by jointly using audio and text representations for more accurate communication aids.$Commercial implications: Enables more precise and reliable speech decoding devices for users with communication impairments by integrating richer brain signal information.
- •For speech technology developers: Create speech recognition systems that incorporate synchronized brain, audio, and text data to enhance understanding of natural speech patterns.
Authors
Suhyun Kim, Jinmo Han, Danny Dongyeop Han, Ahhyun Lucy Lee, Jewoon Lee, Yonghyeon Gwon, Zach Paris, Chun Kee Chung, Saewoong Bahk, Nam Soo Kim, Seong Jae Hwang, Jiook Cha
Abstract
Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio-text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.