Brain conditioned policies improve motor intention decoding accuracy
Brain-Conditioned Action Policies for Neural Motor Decoding
Machine Learning
Summary
People who cannot move their bodies often use brain-computer interfaces (BCIs) to control devices by interpreting their brain activity. This paper introduces BrainVLA, a system that helps translate brain signals into movements by connecting brain data with large language and robotic action models using language as a bridge. The authors trained it to understand neural signals as intentions and then guide robotic actions accordingly. Tests show BrainVLA is better at understanding intended movements across different sessions and needs less training data than earlier methods.
What this means in practice
- •For neural interface developers: Develop motor BCIs that decode intended movements more accurately using pretrained vision-language-action models fine-tuned for neural decoding.
- •For robotic control engineers: Improve robotic device control by integrating brain-derived intention signals with pretrained action policies for flexible motor task execution.
Authors
Luyao Jin, Running Zhao, Huan Zhao, Vincent C. K. Cheung, Wei-Hsin Liao
Abstract
Motor brain-computer interfaces (BCIs) aim to decode motor intention, enabling people with paralysis to control external devices. Neural motor decoding typically learns task-specific mappings from neural activity to kinematics, yet remains constrained by scarce paired neural-action data. We propose BrainVLA, a framework that enables neural motor decoding by drawing on a pretrained vision-language-action (VLA) model through language-mediated alignment. BrainVLA mitigates reliance on scarce paired neural-action data by leveraging VLA policies. We first construct VLA-compatible datasets including paired neural activity, action signals, language instructions, and rendered visual observations. Then, we adapt the OpenVLA-OFT policy to the target action spaces through LoRA fine-tuning. To establish an effective interface through which neural activity can convey motor intention to adapted VLA policies and guide action generation, we train a neural encoder via neural-language alignment, using language representations as semantic targets to capture latent motor intent from neural activity. The resulting neural representations serve as an endogenous intention signal to guide VLA policies to generate executable actions, while visual observations provide complementary information about the evolving task state. BrainVLA is evaluated on two neural motor datasets with different action dimensionalities using causal rollout decoding. It outperforms the evaluated baselines in cross-session decoding $R^2$ and task success rate, while demonstrating high training data efficiency. These results establish a route for neural motor decoding to draw on large-scale robotic priors through brain-conditioned VLA policies.