Automated system extracts speech turns and feedback in conversations
An automated pipeline for standardised speech-unit annotation in spontaneous dialogue
Computation and Language
Summary
Understanding conversations needs clear markers for when people talk and give feedback. The authors created a computer system that automatically finds speaking turns and backchannels in two-person talks recorded on separate channels. Their system uses audio detection, speech recognition, and context checks to mark parts of speech for later human editing. Tested on Danish conversations, it showed moderate accuracy and consistent timing, working similarly even when one person heard noise through special headphones. This tool aims to help make conversation analysis faster and more consistent by providing a first rough pass for human reviewers.
What this means in practice
- •For dialogue system developers: Automatically segment and label turns and listener feedback in conversational data for training and testing dialogue models.
- •For speech data annotators: Provide a consistent automated first pass for marking speech units to streamline manual annotation of spontaneous conversations.
Authors
Hanlu He, Harald Vilhelm Skat-Rørdam, Ingvi Örnólfsson, Ivana Konvalinka
Abstract
Quantifying conversational dynamics requires reliable identification of interactional units and their temporal boundaries, but speech activity alone does not distinguish conversational turns from listener feedback or within-turn pauses. We present an automated pipeline for extracting turns and backchannels from separate-channel recordings of spontaneous dyadic conversation, designed to provide a consistent first-pass annotation for subsequent human review. The pipeline combines voice activity detection, channel-energy filtering, temporal merging, automatic speech recognition, and context-based post-processing. We evaluated the pipeline on 99 ten-minute Danish conversations from 33 dyads using segment-level detection reliability and temporal boundary error. Conversations were recorded under both normal and asymmetric listening conditions. In the latter, speech-shaped noise was delivered to one participant through bone-conduction headphones. Overall detection reliability was F1=0.621, with similar performance for turns F1=0.624 and backchannels F1=0.618. For successfully matched segments, median absolute onset and offset errors were 0.150 and 0.160s for turns and 0.130 and 0.180s for backchannels, respectively. Mean errors were substantially larger for turn boundaries, indicating a smaller number of large boundary mismatches. Performance did not differ significantly across the two experimental listening conditions. In a four-conversation case study, pipeline-human agreement was lower and more variable than human inter-annotator agreement and varied across parameter settings. These results support the pipeline as an automated first pass within a semi-automated annotation workflow, providing a consistent basis for more standardised and reproducible annotation of conversational dynamics.