Audio language models struggle to detect pain from speech tones
Do Audio LLMs HEAR Pain? Tracing Acoustic Evidence from Speech to Language Reasoning
Computation and LanguageArtificial IntelligenceSound
Summary
This study looks at how well special AI models that listen to speech can recognize when someone sounds like they are in pain. The researchers found these models have trouble figuring out pain just from how words sound, doing only a little better than guessing. The models can detect pain when people speak ratings about their pain, but the information about pain fades deeper into the model’s thinking process. When the researchers gave the models clear clues about pain as text, the models did much better at recognizing it.
What this means in practice
- •For healthcare ai developers: Improve pain detection tools by directly incorporating pain-related acoustic features identified in speech as text prompts for better model predictions.
- •For voice user interface designers: Design speech recognition systems aware that current large audio language models may miss subtle emotional or pain-related vocal cues, requiring explicit features.
Authors
Wenqi Li, Xiangjue Dong, Sulaiman Vesal
Abstract
Large audio language models (LALMs) can process speech directly, but it remains unclear how well they use the non-lexical acoustic information that speech carries. We study this question through pain, using TAME-Pain, which contains pain-neutral Harvard sentences and spoken pain ratings recorded at different self-reported pain levels. We introduce HEAR, a four-stage diagnostic framework that traces pain-related information from the speech signal to the model's prediction, and apply it to seven closed-source and open-weight LALMs. On the Harvard sentences, where pain can only be inferred from the voice, all models perform at chance level, and same-speaker references or few-shot examples improve balanced accuracy by no more than six points. On the spoken ratings, the same models reach 87.5-99.7% balanced accuracy. Layer-wise probing of open-weight LALMs shows that pain can be decoded from early audio representations (74.1% balanced accuracy for binary prediction and 45.0% for three-class severity), but that this information, along with most pain-associated acoustic cues, decreases in deeper layers. In an acoustic perception test, weaker models cannot distinguish even large changes in these cues, whereas stronger models can but still fail to infer pain. Finally, we recover pain-related acoustic cues and directly reinject them as structured text into the prompt, which improves pairwise pain prediction by up to 32 points. These results indicate that current LALMs encode paralinguistic information in early layers that is largely unavailable when the model produces its answer.