Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention
2026-06-08 • Sound
SoundMachine Learning
AI summaryⓘ
The authors studied how to detect Parkinson's disease (PD) by analyzing speech, since PD often causes problems with speaking. They combined three different ways to represent speech sounds—spectrograms, MFCCs, and HuBERT embeddings—to capture more information. Their new deep learning model uses a special attention method to smartly mix these speech features and showed strong results on a Spanish speech dataset. Tests confirmed that using multiple types of speech data together helped improve detection accuracy.
Parkinson's diseasehypokinetic dysarthriaspeech analysisLog-Mel spectrogramMFCCHuBERT embeddingsdeep learningcross-modal attentionResNet-18BiLSTM
Authors
George Theodosiou, Loukas Ilias, Dimitris Askounis
Abstract
Parkinson's disease (PD) is a progressive neurodegenerative disorder that frequently causes speech impairments associated with hypokinetic dysarthria. As speech production relies on the precise coordination of complex neuromuscular mechanisms, speech analysis has emerged as a promising non-invasive and cost-effective biomarker for early PD detection. Recent deep learning approaches have shown encouraging results; however, most existing methods rely on a single speech representation, potentially overlooking complementary pathological information encoded across different feature spaces. In this work, we propose a multi-branch deep learning framework for automatic PD detection from speech. Each recording is segmented into 5-second chunks and represented using three complementary modalities: Log-Mel spectrograms, MFCCs, and HuBERT embeddings extracted from raw waveforms. The spectrograms are processed using a pre-trained ResNet-18 encoder, MFCC sequences are modeled through a BiLSTM network, and raw speech is encoded using a pre-trained HuBERT model. To effectively integrate these heterogeneous representations, we introduce a context-guided cross-modal attention mechanism that dynamically weights temporal HuBERT embeddings according to the global acoustic context derived from the spectrogram and MFCC branches. Experiments conducted on the publicly available Spanish PC-GITA corpus under strict speaker-independent 5-fold cross-validation demonstrate the effectiveness of the proposed approach. The proposed architecture achieves an accuracy of 91.51%, an F1-score of 91.24%, and an AUC of 95.97%. Furthermore, ablation studies confirm the contribution of both the proposed context-guided cross-modal attention mechanism and the integration of complementary speech representations. These findings highlight the potential of heterogeneous speech modeling for robust and clinically reliable PD detection.