Summary
Detailed labeling of speech recordings is important for training computers to understand and generate natural speech with emotion and different voices. Existing methods rely on manual work or paid tools, which can be slow, costly, or inflexible. The authors created SpeechAnnotator, a system that uses multiple open-source AI agents working together to analyze speech with its context and check their work for consistency. They also built a set of tests and datasets to measure how well such systems perform. Their approach offers an open, locally usable alternative to commercial systems and improves speech annotation quality by focusing on evidence and context.
What this means in practice
- •For speech technology developers: Train speech models with fine-grained, multidimensional annotations from a local, open-source, and context-aware system that reduces dependency on paid services.
- •For audio data annotation teams: Automate thorough speech annotation workflows using multiple collaborating agents that ensure cross-segment consistency and evidence-based relabeling.
Authors
Qirui Zhan, Shuiyuan Wang, Jingbin Hu, Haoyu Zhang, Xiaming Ren, Jinrui Liang, Chaoren Yu, Bengu Wu, Yunxiang Chen, Houdun Liu, Su Feng, Liumeng Xue, Lei Xie
Abstract
Recent controllable speech generation requires training data with fine-grained annotations of speaker traits, prosody, emotion, paralinguistic cues, acoustic scenes, and context. Existing workflows often rely on manual correction, paid hosted multimodal services, or fixed processing chains, which limits large-scale data processing through annotation cost, external-service dependence, or weak cross-stage recovery. We introduce SpeechAnnotator, a locally deployable, context-aware multi-agent framework built entirely from open-source models and tools. Supporting frontend modules first obtain speaker-aware segments and final segment transcripts, while prior evidence extractors attach heterogeneous segment-level cues. Three specialist agents then collaborate through shared state: the Planning Agent converts local audio evidence, speaker history, neighboring segments, and recording-level context into field-specific contracts; the Labeling Agent performs contract-guided multimodal prediction for directly observable attributes; and the Review Agent runs a bounded review loop that checks evidence support and cross-segment consistency, triggering relabeling only for unsupported or inconsistent fields. To address the fragmentation of existing evaluation resources across isolated tasks and narrow-domain test sets, we introduce SpeechAnnotator-Bench (SA-Bench), containing 8.87 hours of human-annotated audio across nine source formats, together with SpeechAnnotator-Eval (SA-Eval), which separates Timeline-Eval for speaker-aware timeline recovery, Closed-Eval for finite-set attributes, and Open-Eval for open-ended attributes. Experiments and ablations show that SpeechAnnotator provides a locally deployable alternative to commercial audio-capable systems, while the bounded review loop improves multidimensional annotation through evidence- and context-aware field-level recovery.