Language-based models assess counseling quality better using other domains
Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training
Computation and LanguageMachine Learning
Summary
Assessing how well counseling conversations go usually needs experts and lots of data, which is hard to get. This paper shows that language models trained on counseling conversations from other areas can judge new types of counseling better than models trained only on the same kind. The key is that these models focus on the words and structures from expert rating tools, making expert impressions easier to predict. This means what is said carries the expert’s overall judgment more than factors like who is talking or audio quality.
What this means in practice
- •For healthcare quality teams: Improve automatic quality checks in counseling by using models trained on different counseling types rather than limited same-type examples.
- •For call center supervisors: Use language-based assessments drawn from other conversation domains to better judge interaction quality without extensive labeled data in every scenario.
Authors
Tobias Hallmen, Elisabeth André
Abstract
Automatic assessment of communication quality in dyadic counseling conversations is bottlenecked by data: expert-rated corpora are small and expensive to grow. We study cross-domain transfer of expert overall-impression prediction across three German corpora of simulated counseling (two general-practice medical, one school-related parent-teacher; $n=195$ expert-rated sessions, one corpus after scale equating). Training on the other domains beats training in-domain: leave-one-domain-out transfer reaches nested Spearman $ρ= 0.54$ against $\le 0.48$ within the target domain, a paired session-level gap of $+0.15$ that holds at $+0.12$ when the training-set sizes are matched, so it is not simply data volume. The decisive features are session-level construct scores from small open-weight LLMs reading the two-speaker transcript, with the constructs largely derived from the experts' rating instruments: the instrument-derived battery lifts a single judge from $0.32$ to $0.41$ over generic dialogue qualities, judges from three model families ensemble to $0.51$ language-only, and a nonverbal-dyadic block adds $+0.03$ more, not separable from noise at this sample size. We also price the recording setup: one corpus lost its per-speaker audio, 16% of its diarised segments carry the wrong speaker, and repair is worth $+0.07$ there. At practically attainable corpus sizes, the expert's overall impression is carried by what is said, and by other communication programs' data more than by one's own.