Audio models improve with questions and self-directed training rounds
EvoAudio: Recursive Self-Improvement for Audio Understanding
SoundMachine Learning
Summary
Understanding audio is hard because it’s not just about what is said but how it sounds. The authors created EvoAudio, a system that lets an audio model teach itself by making its own training questions and improving step-by-step. Instead of needing people to label sounds, EvoAudio uses tools that know how sounds are made to check answers automatically. This method helped different audio models get better over multiple rounds of practice, boosting their accuracy on several tests.
What this means in practice
- •For audio software developers: Build audio recognition systems that improve quality over time without costly manual labeling by generating training tasks dynamically based on model weaknesses.
- •For speech technology teams: Create speech understanding models that more accurately capture how things sound, enhancing performance on diverse multimodal audio benchmarks.
Authors
Yuxiang Wang, Shengbo Cai, Yingda Shen, Ming-Hao Hsu, Qinke Ni, Liqiang Zhang, Teddy Sun, Steve Yevs, Zhizheng Wu
Abstract
Audio language models understand what is said far better than how it sounds. Closing this gap takes more than data. Detailed acoustic annotation is costly, labels from stronger models inherit their errors and limits, and fixed data cannot adapt as the learner improves. We therefore propose EvoAudio, a recursive self-improvement system for audio understanding. To our knowledge, it is the first to evolve the model, waveforms, questions, and difficulty in one closed loop. EvoAudio uses the current model's performance to set the focus and difficulty of the next training data. A library of audio tools then constructs questions whose answers follow from how the audio was made, providing verifiable supervision without new human annotation. Reinforcement learning updates the model, and validation decides whether it enters the next evolution round. Across 13 rounds, EvoAudio improves five models with different audio encoders and language backbones on MMSU, MMAU-Pro, and MMAR. It achieves the highest average for every backbone, raising overall performance by up to 6.3 points. The improvement unfolds over successive rounds, with each stronger model starting the next round.