Audio driven Cantonese lyric writing tool matches tones to melody
ARIA: Audio-Driven Melody-Tone Relation Modeling for Cantonese Lyric Authoring
Computation and LanguageSound
Summary
Writing Cantonese song lyrics is tricky because the words’ tones must fit the melody’s musical notes. The authors created ARIA, a system that listens to a singing recording and predicts the tones behind the melody, even if the audio is noisy or unclear. It then uses these tone predictions to help generate lyrics that match the melody naturally. They also built a big dataset of Cantonese songs with aligned audio, tones, and lyrics to make this work possible. Tests show their method can accurately understand the melody's tones and write lyrics that fit well.
What this means in practice
- •For music app developers: Create apps that generate Cantonese lyrics from hummed or sung melodies without needing clear musical notation.$Commercial implications: Supports new consumer-facing songwriting apps that convert raw singing input into tone-matched Cantonese lyrics.
- •For language learning platform teams: Develop interactive tools that help learners understand Cantonese tonal patterns by linking singing audio to tone-aware text generation.
Authors
Shengyu Li, Jinting Wang, Li Liu
Abstract
Cantonese lyric writing requires close alignment between lexical tones and melodic pitch. Existing melody-guided lyric generation methods typically rely on symbolic melody to generate lyrics. However, in real songwriting scenarios, melodies are often expressed as raw singing audio or hummed recordings, where pitch is implicit, noisy, and unstructured, making these methods difficult to apply directly. To address this limitation, we propose ARIA, a two-stage audio-driven melody-tone relation modeling framework for Cantonese lyric authoring that generates Cantonese lyrics from singing recordings with provided character-level timestamps. Specifically, we first design a Tri-Stream Relation-Aware Tone Estimator (TRATE) to predict 0243 sequences from timestamped singing audio by modeling multi-stream acoustic cues and relational tonal structure. We then propose a Decoupled Retrieval-Augmented Tone-Conditioned Lyric Generator (DRA-TCLG) to generate fluent lyrics conditioned on predicted tonal plans with retrieval-enhanced lexical guidance. Moreover, we construct a large-scale aligned audio-Jyutping-0243 dataset from real Cantonese singing recordings to support this new task. Experimental results demonstrate that ARIA achieves strong performance in both 0243 prediction and tone-consistent lyric generation, validating the effectiveness of the proposed framework.