Large audio language models improved for precise sound event timing
Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception
SoundArtificial Intelligence
Summary
Finding the exact timing of sounds in an audio clip is hard for many AI models that understand sound and language together. The authors improved such models by adding a dedicated part that looks closely at short moments (frames) in the audio to better locate the start and end of specific sounds. Their approach keeps the main model's ability to understand meaning while improving the timing accuracy. Tests showed that their method works well across various situations and helps explain why the model makes certain timing decisions.
What this means in practice
- •For audio software developers: Create tools that precisely locate sound events within audio for editing or analysis by combining language context with detailed frame-level cues.
- •For security monitoring teams: Improve audio surveillance accuracy by pinpointing exact times of audio events in recordings using the enhanced grounding method.
Authors
Yanfeng Shi, Yan Song, Junhui Li, Tinggan Huang, Wu Guo, Haoyu Song, Ian McLoughlin
Abstract
Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.