Unified Audio Intelligence Without Regressing on Text Intelligence
2026-07-06 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine LearningSound
AI summaryⓘ
The authors created Audex, a new model that understands and works with both audio and text using a single system. They built Audex on a strong existing text-only model and trained it on huge amounts of audio and text data together. This lets Audex handle things like speech recognition, translation, and audio generation very well, while still keeping the smart reasoning abilities of the original text model. The authors also shared the model openly to help further research.
audio intelligencelanguage modelTransformer decodermultimodal generationspeech recognitiontext-to-speechreinforcement learningknowledge distillationMixture of Experts (MoE)audio embedding
Authors
Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
Abstract
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.