HARP: Harmonic-Aware Residual Partitioning for Neural Audio Codecs
Sound
Summary
The authors address a problem in neural audio codecs where all sound frequencies are mixed together, causing unpredictable loss when simplifying data. They propose HARP, a training method that organizes the audio processing into frequency groups, allowing the system to better understand and recreate sounds like overtones with their fundamental tones. This method improves audio quality without changing the model's structure and works better than previous approaches on speech, music, and other sounds. Human listening tests also confirmed the sound is perceptually better.
Authors
Qiaoyu Yang, Lixing He, Binyue Deng, Weifeng Zhao
Abstract
Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.