Deep speech features reveal circular patterns that mark word timing
Deep Speech Representations Track Word Timing along a Low-Dimensional Circular Trajectory
Computation and LanguageMachine Learning
Summary
Understanding exactly when words start and end in speech is important for models that convert audio to text. The authors show that many advanced speech models represent the timing of words in a simple, circular pattern within a small part of their internal data. This pattern is strong enough in one model, HuBERT, to help find where words begin and end without explicit training. Their method can detect word boundaries by tracking these cycles of representation.
What this means in practice
- •For speech recognition teams: Use intrinsic timing patterns in deep speech models to improve word boundary detection in automatic speech recognition systems.
- •For voice assistant developers: Improve the accuracy of voice command parsing by incorporating circular timing trajectories to detect precise word boundaries.
Authors
Burin Naowarat, Hao Tang, Sharon Goldwater
Abstract
Speech timing is fundamental for speech models to recognize words accurately, but how word-timing information is represented in speech models remains underexplored. We study this question using the framework of representational geometry, which investigates whether and how concepts are represented in low-dimensional subspaces of the representation space. Prior work in cognitive neuroscience has shown that when HuBERT feature sequences are averaged across words of the same duration, they form near-circular trajectories from word onset to word offset in a two-dimensional "temporal subspace." Here we show that this circular structure of word timing is a general property of deep speech representations, by demonstrating its emergence in nine additional speech models covering both supervised and self-supervised. We then move beyond averaged word segments and show that in HuBERT, the circular trajectories are consistent enough at the level of individual words to perform word segmentation. Our approach, Rotary Segmentation (RoSeg), achieves 64.3% boundary F-score on Librispeech by detecting word boundaries when cycles complete in the temporal subspace. Although supervision is needed to identify a good subspace, nevertheless this result demonstrates much stronger implicit word knowledge in HuBERT than has previously been shown.