Continuous modeling improves piano music rendering from midi prompts
CPR: Combining global composing, local performing and full-sequence refining in piano rendering with continuous autoregressive modelling
SoundArtificial Intelligence
Summary
Turning piano note instructions (MIDI) into realistic music recordings is tricky because you want to keep both the musical notes and the sound style. The authors introduce a new approach called CPR that combines different techniques: predicting hidden musical information step-by-step, generating sounds locally, and then refining the full sound. This method works directly with continuous sound data, avoiding some limitations of older methods that simplify sound into pieces. It improves the timing and musical details in the generated piano music.
What this means in practice
- •For music production teams: Create realistic piano recordings from MIDI files with accurate musical expression and timbre matching a reference recording.$Commercial implications: Enables music software companies to offer high-quality AI piano rendering tools that produce more expressive and realistic sounds for producers and composers.
- •For audio software developers: Integrate continuous autoregressive models into sound synthesis tools to improve temporal accuracy and detail in instrument rendering.
Authors
Chong Jing, Junan Zhang, Zhizheng Wu
Abstract
Prompt-conditioned piano MIDI-to-Music rendering aims to faithfully render target notes while reproducing the timbre of a reference recording. Existing approaches primarily follow two paradigms: autoregressive (AR) modeling and flow matching (or diffusion). Discrete-codec AR models provide causal temporal modeling, but quantization can discard acoustic detail. Flow matching better preserves acoustic structure in the cost of full-sequence attention costs and worse semantic structure. Continuous autoregressive models operate directly on continuous representations. It not only combines the condition-following ability of AR models and distribution-modeling capacity of flow matching but also bypasses the quantization bottleneck with lower computational costs. Building on this principle, we present Composer--Performer--Refiner (CPR) framework. Composer autoregressively predicts continuous hidden states, Performer generates 24kHz acoustic latents through local flow matching and Refiner then upsamples the waveform to 48 kHz. We further introduce Bottlenecked Representation Alignment (BREPA) and Modality--Time RoPE (MT-RoPE) to strengthen musical semantic structure in Composer hidden states and temporal alignments across modalities. Codes are available at https://github.com/FEAfeatherTHER/CPR_official