Timestamped audio captioning creates detailed speaker timelines
TAC2: Timestamped Audio Captioning as composable functions
Sound
Summary
Audio apps often need to know exactly when different sounds happen and who is making them. The authors created a system called TAC2 that breaks down audio into time-stamped segments with detailed descriptions and speaker info. It learns by combining guesses from different simpler systems to avoid costly manual labeling. TAC2 can then give precise, organized information about sounds and speech, which helps other language models understand audio better or helps generate new sounds with specific timing and speaker details.
What this means in practice
- •For audio analysis developers: Produce accurate, time-stamped transcripts with speaker and speech attribute details from real-world audio recordings.
- •For voice assistant designers: Generate audio responses with precise control over timing, speaker identity, and emphasis on words to improve user experience.$Commercial implications: Enables creation of customizable audio outputs for voice assistants that require fine-grained temporal and speaker control.
Authors
Swapnil Bhosale, Prem Seetharaman, Ke Chen, Haonan Chen, Justin Salamon, Zeyu Jin
Abstract
Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.