Speech llm internal signals detect parkinsons across languages
Exploiting Speech LLM Representations for Multilingual and Cross-Lingual Parkinson's Disease Detection
Sound
Summary
Detecting Parkinson’s disease early through speech is important but challenging, especially across different languages. This paper studies how internal parts of speech large language models (LLMs) can help identify signs of Parkinson’s from voice recordings in multiple languages. The authors find that features from the model’s encoder are more useful than those from the decoder or the final generated text. They also show that combining information from several internal layers improves detection, suggesting important clues are distributed within the model.
What this means in practice
- •For clinical speech technology teams: Create better multilingual tools to detect Parkinson’s disease from voice recordings by using internal speech model features.
- •For developer teams for speech health apps: Improve voice-based health apps’ accuracy by integrating dynamic layer aggregation methods to capture subtle disease indicators in speech.
Authors
Sarthak Giri, Zi Haur Pang, Tatsuya Kawahara
Abstract
Speech Large Language Models (Speech LLMs) have shown strong performance across diverse tasks, yet their utility for pathological speech analysis remains underexplored. In this work, we investigate the effectiveness of internal representations from encoder and decoder components of Speech LLMs for Parkinson's Disease (PD) detection across multilingual and cross-lingual settings. Our findings reveal that encoder representations consistently outperform their decoder counterparts in most models and settings and that pathological cues may be progressively attenuated as audio representations are projected into the language model space. We further show that generative outputs are less reliable for clinical tasks compared to internal representations. To leverage information spread across multiple layers, we propose a Squeeze-and-Excitation (SE)-based dynamic layer aggregation framework, which surpasses best-layer selection in multiple experiments, suggesting that PD-relevant acoustic cues are distributed across transformer layers rather than concentrated in one.