Synthetic whistles and wrong lyrics do not improve real melody search
Whistle and Wrong Lyrics: Synthetic Queries Are Not Enough for Real-World Query-by-Melody
SoundInformation RetrievalMachine Learning
Summary
People often try to find songs by humming, whistling, or singing with wrong words. The authors studied whether training computer systems on fake whistles and silly sung phrases could help find real songs when people whistle or sing wrongly. They found that while training on fake data helps recognize more fake queries, it does not improve finding songs from real whistles or wrong-lyrics singing. This means collecting real examples from people is necessary to improve these systems. The authors will share their code, models, and data.
What this means in practice
- •For music app developers: Develop song search features that rely on real user whistles and wrong-words singing rather than synthetic training data for better accuracy.
- •For audio signal processing engineers: Improve whistle detection algorithms by incorporating real whistle recordings instead of synthetic sounds to enhance retrieval performance.
Authors
Luigi Rivolta
Abstract
People look up a song by humming it, but also by whistling it or singing it with the wrong words. Published query-by-humming systems and benchmarks focus on hums and sung queries, and no public dataset pairs whistles or wrong-lyrics singing with song recordings. We study both query types with a melody-only retrieval system: a key-normalized pitch track embedded by a 5.2M-parameter contour encoder, with a spectral-peak tracker for whistles. First, a physics-grounded generator renders synthetic whistles and nonsense-syllable singing from the vocal melodies of 11,083 songs. Training on them raises top-1 retrieval of held-out synthetic queries against an 11,050-song gallery from 0.180 to 0.251 for whistles and from 0.193 to 0.264 for wrong-lyrics singing, but the gain does not transfer: on the MLEnd test performers the best synthetic-only model gets 210 of 339 real whistles right at top-1, against 214 for the baseline. Second, a closed-set fine-tune on real MLEnd whistles with performer-disjoint splits raises whistle-to-hum top-1 from 0.631 (95% CI 0.581-0.681) to 0.673 (0.622-0.723), on known songs only. Third, two further generations of synthetic queries also fail to transfer: melody-conditioned whistles and la-la singing from a commercial neural music model leave real-whistle top-1 at 0.631, and an open-source pipeline (YingMusic-Singer-Plus singing with partly wrong, mumbled or nonsense lyrics, a DDSP whistle synthesizer, and a melody-error model fitted on real performers) gives at best 0.619 (paired difference -0.012, 95% CI -0.038 to +0.015). Across physics-based, commercial neural and open-source neural synthetic data, nothing transferred to real whistles, which points to collecting real queries as the missing step. Code is public; models and data will be released.