Improving keyword counts from poor police call transcriptions
Conducting Reliable Keyword Estimation with Unreliable ASR in Police Audio Data
Computation and LanguageArtificial Intelligence
Summary
It is expensive and slow to have humans transcribe large amounts of police radio audio. Automatic systems that transcribe speech can be faster but often make mistakes, especially with police-specific words. The authors tested a method that combines automatic transcriptions with some human-labeled examples to get better, more reliable counts of important words. They found this method works better than using either automatic or human transcriptions alone for analyzing police conversations in Baltimore and Chicago. This approach helps get trustworthy information from noisy transcripts without needing to transcribe everything by hand.
What this means in practice
- •For police data analysts: Obtain more reliable keyword frequency estimates from police radio audio with minimal human transcription effort.
- •For law enforcement software developers: Improve processing of police audio transcripts by correcting systematic ASR errors to better support operational insights.
Authors
Robert Cole Molloy, Rhiannon N. Miller, Stephen Morgan, Anjalie Field
Abstract
Extracting insights from large audio data sets using human transcription is expensive and time-consuming. Off-the-shelf Automatic Speech Recognition (ASR) systems offer an easy-to-use scalable alternative, but systematic transcription errors, especially in domain-specific settings, can bias downstream analyses and lead to unreliable conclusions. We investigate whether imperfect ASR transcripts can still be used to obtain statistically valid keyword frequency estimates by combining model outputs with a small set of human-labeled examples, following Prediction Powered Inference and Design-based Supervised Learning frameworks. We evaluate this approach on police radio communications from Baltimore and Chicago, a domain in which keyword distributions could provide valuable insights into police activity, but off-the-shelf ASR systems exhibit systematic errors on operationally important terminology. Using bootstrap-based confidence interval coverage as our evaluation metric, we find that our approach substantially increases the number of domain-specific keywords that can be analyzed with statistically valid confidence intervals, from 43.0% using ASR alone and 27.2% using only a small set of human-labeled transcriptions to 56.2% in the Baltimore dataset, with similar improvements observed in Chicago data. Our results demonstrate that post-hoc error correction can improve statistical validity of downstream analyses even for poor-quality transcriptions, offering an avenue for better leveraging off-the-shelf ASR models for large-scale speech analysis.