Levantine Arabic audio benchmark enables better speech technology evaluation
SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic
Computation and Language
Summary
Levantine Arabic is spoken by many people but has no standard spelling, making speech technology hard to test. The authors created SHAMS, a dataset of 1,300 recordings from five regional dialects with matching text and sound details. This helps developers check how well their tools understand and transcribe Levantine Arabic speech. They tested existing speech models on SHAMS to show how it can measure progress in this area.
What this means in practice
- •For speech technology developers: Evaluate and improve Levantine Arabic speech recognition and transcription systems using a balanced multi-dialect audio-text benchmark.
- •For language learning app makers: Test pronunciation and diacritization accuracy across regional Levantine dialects to build better language learning tools.
Authors
Ben Sapirstein, Roy Mattar, Guy Mor-Lan, Ahlam Mohamed, Letizia Cerqueglini, Morris Alper
Abstract
Levantine Arabic (LA) is spoken by tens of millions of people, creating a pressing need for shared benchmarks to evaluate LA speech-language technologies. Evaluating such technology is particularly challenging given LA's internal diversity and its opaque and non-standardized orthography. We present SHAMS (SHami Annotated Multi-dialect Speech), a benchmark comprising 1,300 utterances drawn from open audio corpora, balanced across five LA varieties (Urban and Rural Palestinian, and Urban Jordanian, Lebanese, and Syrian). Each utterance is represented across four aligned tiers: audio, unvocalized orthography, diacritized text, and phonetic transcription. This structure supports evaluation of various downstream tasks such as diacritization, grapheme-to-phoneme conversion, automatic speech recognition, and audio-to-phoneme, grounded in audio and stratified by variety. We benchmark open and proprietary models across these tasks to demonstrate the utility of this benchmark for measuring progress across LA. We release SHAMS at https://shams-nlp.github.io .