Japanese speech recognition improves reading precision with ruby annotations
Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition
Computation and LanguageArtificial Intelligence
Summary
When computers listen to Japanese speech and write it down, they usually only record the usual spelling of words. But in Japanese, the same letters can be said in different ways, which computers often miss. The authors offer a new method called Ruby-ASR that links each written part to how it’s actually spoken, helping computers understand both. Their tests show this approach helps recognize the exact pronunciation without losing the standard spelling. They also provide tools so others can use their improvement.
What this means in practice
- •For speech recognition developers: Improve Japanese ASR systems by embedding local pronunciation details directly in transcription output for more accurate reading recognition.
- •For subtitle production teams: Create subtitles with linked pronunciation metadata, enabling clearer reading distinction in Japanese video captions.
Authors
Hao Shi, Yun Liu, Xuehao Yang, Jun Liu, Chuanbo Hua, Xuanjun Chen, Lianbo Liu, Shiao Zhu, Zixiong Su
Abstract
Conventional Japanese automatic speech recognition (ASR) is supervised by an orthographic transcript, although the same written form can correspond to different lexical readings realized in speech. Such utterances receive an identical target, so their reading distinction is absent from the supervision interface and cannot be recovered reliably by post-hoc text-only grapheme-to-phoneme conversion. We present Ruby-ASR, which refines the conventional target into a span-bound orthographic--lexical-reading sequence. Unlike separate full-sentence orthographic and phonological outputs, the ruby representation locally binds each written span to its realized reading and permits deterministic recovery of both views. We instantiate the target under subtitle-style and verbatim-style transcription conventions using a Qwen3-ASR backbone; a mora-level CTC objective provides auxiliary monotonic reading supervision. The experimental results across five Japanese benchmarks show that refining the recognition target can improve lexical-reading recovery without sacrificing readable orthographic transcription. We release the checkpoints and inference code.