Multimodal language models struggle to assess psychotherapy skills accurately
MIS-Bench: Benchmarking Multimodal LLMs for Psychotherapeutic Interpersonal Skills Assessment
Computation and Language
Summary
Assessing how therapists interact is important but hard to do automatically. The authors show that current multimodal language models, which analyze videos and text, do not match experts well when rating therapeutic interpersonal skills. They create a new set of therapy video examples with detailed human ratings to test these models. They also develop a special fine-tuning method that helps models improve their scoring accuracy, though there is still a big gap compared to expert judgment.
What this means in practice
- •For mental health technology developers: Create tools that automatically evaluate therapists’ interpersonal skills from recorded sessions using improved multimodal models trained on expert ratings.$Commercial implications: Enables development of therapist training and quality assurance products that rate interpersonal skills more reliably using video and language analysis.
- •For human resources teams: Use multimodal models to assist in evaluating communication and interpersonal abilities in video interviews, informed by therapist skill assessment techniques.
Authors
Yuhan Lu, Yi Yao, Hua Shen, Katie Aafjes-van Doorn, Zhaonan Wang
Abstract
Multimodal large language models (MLLMs) are increasingly used as evaluators, yet their reliability in professional assessment tasks that require expert judgment remains unclear. We investigate this challenge in the context of assessing psychotherapeutic interpersonal skills and introduce MIS-Bench, a Multimodal Interpersonal Skills (MIS) benchmark comprising 996 psychotherapy response videos annotated across 8 dimensions of Facilitative Interpersonal Skills. Across 9 MLLMs with multiple modality and prompting settings, we find that current models show only modest agreement with human experts, inconsistent gains from multimodal input, and limited benefits from reasoning-based prompting. To mitigate this gap, we propose MIS-RAFT, a regression-aware fine-tuning method inspired by RAFT and tailored to fine-grained interpersonal skill scoring at one-decimal precision. MIS-RAFT addresses the mismatch between autoregressive token prediction and scalar-valued expert assessment, significantly improving agreement with human ratings. Overall, MIS-Bench reveals a clear gap between general multimodal capability and expert-level interpersonal judgment, while MIS-RAFT offers a promising path toward more reliable model-based assessment.