Papers for

brain-computer interface teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Brain audio and text signals aligned for better speech decoding

BAT-CLIP: Trimodal Alignment of Brain, Audio and Text

Abstract: Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio-text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.

Fri 25 SeptSoundArtificial IntelligenceMachine Learning

The gist

Decoding speech from brain signals usually links brain activity to either sound or text, but not both at the same time. The authors developed BAT-CLIP to connect brain signals with both audio and text together, which better reflects how the brain processes speech. Their method produces stronger and more informative brain signal representations than earlier two-way methods. They also show that using models trained without explicit labels helps this alignment work better.

Open → 2609.31180v1