Audio language models get new test for true reasoning skills
Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Sound
Summary
Large audio language models can identify sounds and answer questions, but their high scores might come from guessing instead of real understanding. The authors created ARIA-Rubrics, a new way to check if these models really think through audio problems correctly by looking at the whole reasoning process, not just the final answer. Their approach doesn’t need expensive human checks and works by breaking down the models’ thinking steps. Tests showed ARIA-Rubrics aligns well with how humans judge true reasoning and revealed different thinking styles in current models.
What this means in practice
- •For audio system developers: Use ARIA-Rubrics to improve audio reasoning components by reliably measuring genuine understanding versus guessing.
- •For voice assistant engineers: Deploy ARIA-Rubrics to make voice assistants better by verifying their reasoning process on audio commands beyond simple accuracy.
Authors
Yupei Li, Qiyang Sun, Mohamed Mady, Chenxi Wang, Zhengwei Gong, Berrak Sisman, Björn Schller
Abstract
Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.