Multimodal method links speech to picture details for dementia detection
NarraSpace: Multimodal Scene-Grounded Evidence for Picture-Description Dementia Classification
Computation and Language
Summary
People with dementia often have trouble describing pictures, and this can help tell if someone has the disease. The authors developed a tool that connects what someone says and when they say it to specific parts of a picture they describe. The tool then looks at how the descriptions differ between people with Alzheimer’s and healthy people. This helps not just to predict if someone might have dementia but also to show exactly which parts of the picture and speech led to that prediction.
What this means in practice
- •For clinical speech analysts: Use scene-linked speech analysis to support diagnosis by highlighting specific speech and picture elements related to dementia.
- •For healthcare technology developers: Incorporate multimodal dementia detection that ties speech transcripts to picture events into digital diagnostic tools.$Commercial implications: Enables development of commercial software aiding clinicians with explainable dementia diagnostics based on spontaneous speech and visual stimuli.
Authors
Qingming Li, Ruiyan Sun, Satoshi Nakamura
Abstract
Picture-description dementia classification uses a shared visual scene, but a transcript-level prediction alone does not identify the scene evidence underlying its inputs. NarraSpace links words and speech times to named picture elements and retrieves utterances for predefined scene events. Narrative-spatial descriptors summarize coverage and temporal organization. A frozen language model contrasts each event's text under examples from Alzheimer's disease and healthy-control training participants, alongside transcript surprisal and coverage measurements. Retained source records link event scores to their source utterances, speech intervals, and lexical matches. On the official 48-participant test set of the Alzheimer's Dementia Recognition through Spontaneous Speech benchmark, a fixed support vector classifier achieves $80.88\%\pm2.56\%$ mean accuracy over fifty seeds; averaged probabilities yield $81.25\%$ accuracy and $91.32\%$ area under the receiver operating characteristic curve. NarraSpace makes the construction of scene-linked classification inputs explicit and inspectable by connecting event-indexed language-model measurements to their source utterances, speech timing, and extraction records. Such source-linked measurements could support clinician review of picture descriptions by presenting quantitative summaries alongside the original speech evidence.