Mizar small audio language model improves sound understanding on CPUs
Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding
SoundComputation and Language
Summary
Understanding audio with computers usually needs big models that require lots of memory and power. The authors created Mizar, a smaller audio-language model with about 159 million parameters, which is easier to run on everyday devices. They combined different training steps to help the model understand sounds and answer questions about them accurately. Mizar works well on standard tests and can run quickly on just a single CPU without special hardware.
What this means in practice
- •For mobile app developers: Create apps that understand and answer questions about sounds quickly on phones without needing cloud support.$Commercial implications: This paper enables efficient audio question-answering apps to run locally on mobile devices with limited memory and processing power.
- •For home automation engineers: Integrate sound recognition that explains audio context directly on smart home devices with limited hardware.
Authors
Kaiyang Li, Shaobo Han, Yue Tian, Shihao Ji
Abstract
Audio-language models (ALMs) integrate acoustic perception with the knowledge encoded in language models, enabling contextual understanding of auditory events. Making these capabilities practical on devices with limited memory and computation motivates our focus on small ALMs with fewer than 200M parameters. We introduce a recipe that brings together architecture, data, and three-stage training to build Mizar, a 159.3M-parameter ALM. Its architecture connects a compact CED-Small audio encoder to SmolLM2-135M through a frequency-merging mapper. With supervision drawn from ReasonAQA, AudioMCQ, and AVQA, the model undergoes three training stages: audio-language alignment (Stage 1), audio-dependent fine-tuning (Stage 2), and post-training (Stage 3) aimed at strengthening weak skills while retaining learned capabilities. Across five random seeds, Mizar achieves mean accuracies of 52.92% on MMAU, 42.42% on MMAR, and 36.02% on ADQA-clean, surpassing the previous best-performing ALM below 200M parameters on all three benchmarks. It also supports local inference on a single CPU: on questions from the MMAU benchmark, the mean latency from opening the audio file to generating a complete answer is 1.09 seconds. Code and checkpoints are available at https://github.com/KaiyangLi1992/Mizar_159M.