Audio equalization improved by direct preference density alignment method
Direct Preference Density Alignment for Conversational Audio Equalization
SoundArtificial Intelligence
Summary
Training AI models to adjust audio quality usually involves complicated reward systems that can be unstable and large. The paper’s authors propose a new approach that skips the need for these reward models by using large amounts of user preference data. This method combines strengths of online learning and offline refinement, leading to better performance in tuning audio using less computing power. In tests, their approach helped a 1.5 billion parameter AI model match the audio quality of a stronger baseline while being more efficient.
What this means in practice
- •For audio software developers: Develop audio equalization tools that achieve high perceived sound quality using less compute during inference.
- •For voice assistant engineers: Enhance real-time conversational audio quality tuning using preference data without expensive reward models.
Authors
Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe, Pablo Martinez Nuevo, Zheng-Hua Tan
Abstract
Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute.