Multimodal signals reveal self-compassion during reflective interviews
Multimodal Detection of Higher-Order Behavioral Constructs: Self-Compassion in Structured Reflective Interaction
Machine LearningComputation and LanguageComputer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Many feelings and qualities, like how kind people are to themselves when things go wrong, can't be seen directly but can be guessed by how they talk and move. The authors studied self-compassion during special interviews where people talk about tough experiences. They recorded video, audio, and text, then combined these to detect different states of self-kindness or self-criticism. They found that using all types of data together gave better results than any single type alone. This work helps show how complex feelings can be understood with technology.
What this means in practice
- •For mental health app developers: Integrate signals from video, audio, and text to detect self-compassion during user reflections in therapy or coaching apps.$Commercial implications: Enables mental wellness apps to provide personalized feedback by identifying emotional states related to self-kindness, enhancing user support.
- •For human resources trainers: Use multimodal monitoring during structured interviews to assess employees’ emotional regulation and self-compassion in difficult conversations.
Tested on one dataset.
Authors
Siddhant Jain, Dimitra Tsovaltzi
Abstract
Many of the qualities that matter most in how people learn and grow, how someone regulates their emotions, reflects on a setback, or stays aware of others during a difficult conversation, are not directly observable. They have to be inferred from how someone speaks, moves, and sounds over time, and they resist the kind of clean labeling that most machine learning pipelines are built around. We study this challenge through a case that is well grounded in psychological theory but rarely modeled computationally: self-compassion, the tendency to respond to one's own setbacks with patience rather than harsh self-criticism. We examine how it appears during structured reflective interviews in a technology-mediated training setting, where people naturally talk through socio-emotionally demanding situations. Since no existing dataset captures this kind of construct in this kind of setting, we collected and annotated 51 reflective dialog sessions using an independent, temporally overlapping annotation scheme grounded in established theory. We consolidate the underlying six-component psychological model into a three-class supervision space, balancing self-kindness and mindfulness against self-critical or overwhelmed states, and build a reproducible window-based pipeline that aligns video, audio, and text on a shared timeline. Unimodal models trained on each modality separately are compared against a simple probability-level fusion strategy, which yields modest but consistent gains over the best single modality. We close by discussing where each modality succeeds or struggles, what this suggests about how this kind of construct is actually expressed in reflective speech, and what would be needed to model it, and constructs like it, more effectively.