Language supervision improves neural decoding across brain signals
Teaching neural decoders with language
Artificial Intelligence
Summary
Brain-computer interfaces read brain activity to understand thoughts or commands, but teaching these systems usually only shows which class the signals belong to without explaining their relationships. The authors propose using natural language descriptions as supervision to help the system learn better connections between signals and their meanings. They tested this approach on different types of brain signals related to vision, movement, and speech, and saw improvements, especially for decoding speech. This method also helps the system adapt to new subjects and situations more easily and understand new stimuli without retraining.
What this means in practice
- •For brain-computer interface engineers: Enhance speech decoding systems by aligning brain signals with natural language descriptions to improve accuracy and generalization.
- •For neural signal processing teams: Develop adaptable decoding models that transfer across subjects and sessions using text-based supervision for brain signal interpretation.
Authors
Xuan Wen, Yonghao Song, Yuhong Zhang, Guoliang Li, Tzyy-Ping Jung, Xiaorong Gao
Abstract
Brain-computer interfaces decode neural activity, yet class-label supervision (CLS) leaves relationships among classes implicit. We propose Language as Supervision (LaS), which aligns neural features with natural-language class descriptions through learned projections and predicts classes using candidate text prototypes. We evaluate seven encoder architectures with independently trained models for visual, motor, and speech decoding from scalp and intracranial recordings. LaS improves speech-decoding accuracy by up to 17.15~pp over matched CLS baselines. Its representations support candidate-set reuse and label-supervised subject and session adaptation, while LaS enables conditional unseen-frequency decoding in steady-state visual evoked potential (SSVEP) responses. Correct class-description correspondence supports this generalization, and dual-sided attribution visualization shows neural and textual contributions to the same class-matching score. These findings suggest that text descriptions can serve as general-purpose references for learning and interpretation in neural decoding.