Text to MIDI generation gains control with sparse autoencoder features
MI-MIDI 2: From Observation to Control with Sparse Autoencoders in Text-to-MIDI Generation
SoundArtificial Intelligence
Summary
Creating music from text is tricky because it’s hard to control which parts of a computer’s internal model change the sound. The authors studied a type of model called sparse autoencoders to see if the features they find truly influence the music generated from text. They developed a testing method that checks if these features really steer aspects like the type of instruments or note density. Their tests showed some features do meaningfully control what music the system creates, making it easier to guide music generation.
What this means in practice
- •For music software developers: Add controls to text-to-MIDI tools that steer music features like instrument type based on validated internal model features.
- •For audio plugin creators: Design plugins that monitor and adjust MIDI generation in real time using trustworthy sparse autoencoder features for creative control.
Authors
Jakub Poćwiardowski, Mateusz Modrzejewski
Abstract
Sparse autoencoders (SAEs) can expose features of a generative model without predefined labels, but in language models the evidence used to interpret these features has proven an unreliable guide to which of them steer the output. Whether interpretations of SAE features predict control in music generators has not, to our knowledge, been tested. We develop an end-to-end protocol for discovering and evaluating SAE features in text-to-MIDI generation and apply it to text2midi and MIDI-LLM. The protocol characterizes features through correlations with musical metrics, shared instrument programs, and automated descriptions, each with its own control: note-density adjustment, matched inactive excerpts, and permutation calibration. It then tests candidates by intervention, measured directly in the generated MIDI against unmodified generation and a matched random direction, ranks them in screens, and confirms selected features on new prompts. Evidence that survives these controls points to features that steer the output. Descriptions exceed permutation controls for instrumentation, and a feature whose description names a synth-lead program introduces that program in a prespecified confirmation and in a replication on disjoint prompts and reappears in independently trained dictionaries. Features selected by density-adjusted correlations show a small positive shift of their target metric as a family, whereas features selected by raw correlations show none. An atlas provides MIDI, piano-roll, and audio examples for every feature of the targeted screen.