CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning
2026-07-06 • Sound
SoundComputation and Language
AI summaryⓘ
The authors introduce CARD, an audio captioning model that does not use an audio encoder during its actual use, making it more efficient. Instead, CARD trains a small component to connect a large language model with useful audio information learned from a teacher model. They found that sending different types of audio knowledge to different parts of the model improves caption quality. Their approach gets closer to the performance of traditional models that keep the encoder, but without needing the encoder during inference.
audio captioningencoderlarge language modelLoRA adaptersknowledge distillationCLAP-HTSATCIDEr-DAudioCapsClotho datasetperceptual vs semantic features
Authors
Ganesh Pavan Kartikeya Bharadwaj Kolluri, Yuchen Zhang, Michael Kampouridis, Ravi Shekhar
Abstract
Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.