Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification
2026-08-10 • Computation and Language
Computation and LanguageSound
AI summaryⓘ
The authors explored how an audio-based model called PhonoQ, which learns specific speech sound features, can help understand mouth movements during speaking captured by real-time MRI. They combined PhonoQ's audio features with existing models to better identify detailed speech sounds and features like voicing and place of articulation. Their approach improved accuracy in recognizing different speech sounds, even for new speakers or unheard speech. They also found that audio information can help improve models that only use movement data. Finally, their analysis revealed speech patterns that match known ways sounds change in casual speech.
Real-time MRIPhonological featuresArticulatory modelingConformer moduleManner of articulationPlace of articulationVoicingPhoneme classificationSpeech acoustics
Authors
Abner Hernandez, Tomás Arias Vergara, Daiqi Liu, Andreas Maier, Paula Andrea Pérez-Toro
Abstract
Real-time MRI makes it possible to observe vocal-tract articulation during speech, but mapping these articulatory patterns to phonetic and phonological categories remains challenging. We investigate whether PhonoQ, an audio-based model trained to recognize structured phonological features, provides useful information for audio--articulatory modeling. Specifically, we extract representations from PhonoQ's Conformer module, whose training is shaped by supervision for manner, place, voicing, and vowel features. Using articulatory contours with synchronized audio-derived features, we compare WavLM-large and HuBERT-large baselines with models that incorporate PhonoQ-derived representations. Across unseen-speech and unseen-subject settings, these features improve macro-F1 for phonological targets including manner, place, voicing, vowel height, and vowel backness, and also improve fine-grained 39-phoneme classification. In a contour-only inference setting, audio-derived teacher supervision yields modest but consistent gains over contour-only training, indicating that phonological information from synchronized audio can be partially transferred to articulatory models. Finally, posterior analyses show interpretable surface-sensitive patterns consistent with flapping-like /t/ realizations, /t/-/r/ retraction or affrication, and nasal place assimilation.