Music generation improved with graded rewards for following constraints
Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generation
SoundArtificial Intelligence
Summary
Making computer-generated music follow many user rules at once is hard, and current methods often fail when multiple rules combine. The authors created a test with many music rules and showed this problem clearly. They designed a new training method that rewards following each individual rule, not just all at once, helping the model learn better. This approach improves music generation quality without extra human help and works even on new rule combinations.
What this means in practice
- •For music technology developers: Build music generation systems that better follow multiple detailed user constraints using graded rewards without extra human annotation.
- •For automated content creation teams: Generate music reliably meeting complex requirements for media, games, or advertising by combining verifiable programmatic constraints.
Authors
Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoyu Ma, Haoran Shou, Xiaoying Tang
Abstract
Constraint-following music generation asks a score to satisfy several user-specified properties at once, each checkable programmatically (key, meter, length, range, final note, rhythm, motion and form), yet no existing benchmark isolates this capability. We construct MusicConstraintBench, 2,180 items over eight constraint families, on which current models fail once a few constraints are combined. The natural remedy is reinforcement learning with these verifiers as reward, yet we observe that a reward paid only when every property holds leaves most training groups without a learning signal: over the first 50 updates, 0.550 of rollout groups score identically and receive no gradient, even though a failing score typically misses only one requested property. Under the joint criterion, rollouts for a prompt tend to fail together, so a binary reward cannot separate a nearly correct score from a malformed one. We therefore introduce MusicRLVR, which pays graded per-property credit behind a hard validation gate that rejects malformed outputs, plus a joint-satisfaction bonus, requiring no human annotation, learned reward model, or music-domain fine-tuning. On MusicConstraintBench, MusicRLVR lifts Qwen3-4B-Instruct from 0.160 to 0.807 on mixed constraints and leads every zero-shot baseline including Llama-3.1-70B at 0.380. It also generalises to property combinations unseen in training and to out-of-range parameter values, showing that verifiable rewards need not presuppose a target output.