Human annotation defines errors in Quran recitation transcripts
What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts
Computation and Language
Summary
Checking mistakes in Quran recitations using automatic speech transcripts is tricky because some errors look like normal repetitions or accepted spelling differences. The authors created detailed human annotations to label different event types in 100 recordings and tested several algorithms to score these. They found that distinguishing true mistakes from other events depends on clear rules about error labeling and span boundaries. This work helps improve how automatic systems understand Quran recitations by clarifying what counts as a mistake versus other speech events.
What this means in practice
- •For speech technology developers: Improve automatic Quran recitation scoring by integrating annotated event distinctions between mistakes and repetitions.
- •For language data annotators: Use the provided annotation framework and evaluation tools to reliably label and assess transcript errors in religious or formal speech datasets.
Authors
Mohamad Al Mdfaa, Nursultan Askarbekuly, Ahmed Helaly, Ubai Sandouk, Manuel Mazzara
Abstract
Checking Quran recitation from an ASR transcript requires distinguishing unresolved mistakes from repetitions, repairs, opening formulas and accepted spelling differences. We report a completed human annotation of 100 production recording cases: 348 scored units and 162 localized events across ten combined labels. An executable evaluator scores labels and word positions together. A plain diff reaches label-aware F1 0.525 and localization F1 0.826; adapted production cleaner/alignment components reach 0.518 and 0.786, with exact-span F1 0.505 for both. Correcting the adapter's word coordinates recovers all five annotated repetition events, showing why annotation interfaces must be checked before interpreting baseline failures. In a preliminary pilot, eight single 20-minute runs across three coding agents and eight models span label-aware F1 0.143 to 0.892: seven land far above every baseline, and one collapses below the naive diff from a missing normalization step. Across the six, 970 of 972 gold-event instances draw an overlapping prediction, so what remains is not detection but convention: span extent, and the labels whose boundary is stipulated by adjudication rather than visible in the text. Seven of 162 events defeat all six same-day runs, five of them one orthographic rule, and the strongest run still misses the same ones. No run annotated before building, so the pilot measures the algorithm half of the task only.