Papers for

audio analysis developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Timestamped audio captioning creates detailed speaker timelines

TAC2: Timestamped Audio Captioning as composable functions

Abstract: Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.

Tue 27 OctSound
The gist
Audio apps often need to know exactly when different sounds happen and who is making them. The authors created a system called TAC2 that breaks down audio into time-stamped segments with detailed descriptions and speaker info. It learns by combining guesses from different simpler systems to avoid costly manual labeling. TAC2 can then give precise, organized information about sounds and speech, which helps other language models understand audio better or helps generate new sounds with specific timing and speaker details.
Open → 2610.36568v1