Transformer models reveal and control music harmony internally
Harmonic Representations in Symbolic Music Transformers are Localized and Controllable
SoundMachine Learning
Summary
Transformer models trained to create music show signs of understanding the harmony and key changes within the music they generate. The authors studied a music transformer on Bach chorales and found that the model’s middle to later layers hold key information about the music’s mode and key. By probing these layers, they could both identify and change harmonic patterns during music generation, effectively steering the model’s output toward different musical resolutions. This means the model’s internal understanding of music can be used to control how it composes in real-time.
What this means in practice
- •For music software developers: Build music generation tools that directly steer harmonic progressions by manipulating internal model representations at inference time.
- •For interactive art creators: Create interactive music systems where users control harmonic elements of generated music using the model’s learned internal knowledge.
Tested on one dataset.
Authors
Adi Silberschein, Megan Wei, Tamar Rott Shaham
Abstract
Transformer models trained on symbolic music have become remarkably capable. However, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally "know" about its harmonic context, and can that knowledge be used to affect the generation? Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to harmonic changes reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality. This demonstrates that, at inference time, the model's internal harmonic knowledge can be directly manipulated to control the harmonic output.