Policy labels for music requests often ignore exact song names
When the Label Ignores the Request: Auditing Policy-Selected Targets in Synthetic Conversational Music Recommendation
Information Retrieval
Summary
Often, when a music recommendation system is tested using conversations generated by large language models, the label (or answer) chosen by the system doesn't match the exact song a simulated user asked for. The authors found this mismatch happens about half the time when the user names a specific song. This matters because in real music searches, people expect exactly the song they ask for, and alternatives can reduce satisfaction. The authors showed that adding a small amount of extra training data that aligns labels with exact song requests improves system accuracy without hurting overall metrics.
What this means in practice
- •For music streaming platform developers: Improve recommendation accuracy by adding exact-song request data to training to better satisfy users searching for specific tracks.
- •For voice assistant engineers: Enhance voice-based music search by ensuring system responses respect exact user song names, reducing user dissatisfaction.
Authors
Sanjeev Suresh
Abstract
Synthetic dialogues generated by LLM pipelines now serve as complete conversational-recommendation benchmarks: an LLM listener talks to an LLM recommender, and the track logged next in the conversation becomes the official label for each turn. These policy-selected labels make large-scale evaluation reproducible, but they are proxies for what the simulated user asked. We audit the one place where label and request are directly comparable: turns where the user asks for an exact song by name. In the RecSys Challenge 2026 TalkPlay benchmark, using visible dialogue and catalog metadata alone, we find that the official label contradicts the user's exact-song request in half of the audited development turns. This matters beyond one benchmark: naming the desired item is the dominant intent in real music search, where deployed systems avoid substituting an alternative for an exactly named item, on the premise that it costs satisfaction. A small training-time supplement closes most of the gap: adding catalog-resolved request-satisfying targets to a small fraction of training turns yields a 53.3% relative gain in nDCG@20 on the 43 conflict turns while leaving the official metric intact, verified against a matched control that detects the same requests but trains only on official labels.