Neuropsychological label recalibration improves speech based dementia screening accuracy
Improved automatic speech-based cognitive assessment using neuropsychologically calibrated diagnostic labels
Sound
Summary
Diagnosing cognitive problems like dementia using speech is tricky because the labels used to train computer models are sometimes unreliable. The authors improved the labels by adjusting them with brain test results measuring memory and thinking skills. When they trained speech recognition models with these refined labels, the models became better at identifying cognitive issues both in their dataset and in other public ones. This shows that cleaning up the training labels is more important than changing the model itself for better dementia screening.
What this means in practice
- •For healthcare app developers: Create speech analysis tools for early dementia detection with improved diagnostic label quality based on neuropsychological calibration.$Commercial implications: Enables the development of commercial cognitive health apps with higher accuracy dementia screening from speech data.
- •For clinical research teams: Use recalibrated diagnosis labels to improve automatic cognitive impairment detection models trained on real-world speech data.
Authors
Madhurananda Pahar, Caitlin Illingworth, Adam Hampshire, Peter J Hellyer, Agata J Hampshire, Daniel Blackburn, Annalena Venneri, Heidi Christensen
Abstract
Although automatic detection of cognitive impairment from conversational speech is a promising way, it becomes less accurate, as large real-world datasets often contain unreliable diagnostic labels due to self-referral and limited clinical verification, and here we investigated whether neuropsychologically calibrated diagnostic relabeling improves it. We analysed 138 hours of conversational speech from 720 participants, recalibrating unreliable diagnostic labels using attention, executive function, and memory performance via external clinical thresholds, Gaussian, and normative z-score calibration. Transformer-based classifiers were trained using both original and recalibrated labels and evaluated internally on 150 clinically verified participants and externally on Pitt, ADReSS and PROCESS-2 datasets. Memory- and MoCA-based z-score calibration produced the greatest improvements. Using our internal dataset, $F_1$ increased from $0.70$ to $0.83$ ($+13\%$) and from $0.54$ to $0.66$ ($+12\%$) for binary and multiclass classification, respectively. For external validation, the increases are $+4\%$, $+5\%$ and $+9\%$ on Pitt, ADReSS and PROCESS-2 corpus. Neuropsychologically calibrated relabeling substantially improves speech-based cognitive assessment, highlighting label quality, rather than model architecture, as a critical bottleneck in clinical AI for dementia screening.