Transformer models improve music prediction mainly by copying repeats
How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models
Sound
Summary
This study looks at how far back in a piece of music a Transformer model needs to look to make good predictions about what comes next. The authors find that much of the improvement from using long context comes from the model copying repeated segments in the music. When these repeated parts are removed or changed, the benefit largely disappears. This effect is especially strong in folk tune datasets but smaller in more varied datasets like classical performances. The paper also highlights some common mistakes to avoid when measuring how much context helps models.
What this means in practice
- •For music software developers: Build music generation tools that focus on efficient long-range memory to handle repeats and improve prediction quality in symbolic music.
- •For audio performance analysis teams: Improve analysis models by tuning context length based on repetition patterns in performance and score data.
Authors
Amir Fathi
Abstract
We investigate how predictive performance depends on the maximum context available to an autoregressive model of symbolic music, and what information long-context models exploit. A small causal Transformer is trained separately at each context length T in {6, 18, 48, 96, 192, 336} and evaluated on the same target positions, both over raw tokens and over non-overlapping binary latent codes. On Nottingham folk tunes, the token predictor's test NLL decreases by 71% (0.986 bits per token) between 6 and 336 tokens, and by 71% on the O'Neill's tunes long enough for the same sweep (129 of 302 test tunes). Most of the long-context gain is explained by exact copying: the improvement appears when an earlier occurrence of the target's 16-token history enters the available context; overwriting that occurrence removes the gain, whereas equally large unrelated corruption does not; and a simple copy baseline recovers 94% of the reduction. A trained 336-token model likewise loses most of this gain when its history is restricted to recent tokens at test time. Because many repeats arise from written repeat signs expanded in the score, this result pertains specifically to these rendered score representations. By contrast, on MAESTRO performances and MusicNet scores, where exact repeats are substantially less frequent, the reduction is smaller (9% and 18%) and is largely attained by 96 tokens. Finally, an audit of an earlier draft that reported saturation at 16 tokens identified split leakage, overlapping latent receptive fields, an averaging predictor, and a per-file tempo grid; we present these as methodological checks for context-length measurements.