Human verified AI cuts manual time in environmental sound labeling
A Human-Verified AI Workflow for Annotating Environmental Sound Data
Machine LearningSound
Summary
Labeling sounds from nature can take a lot of time, especially when some sounds are rare and data is partly labeled. The authors built a system where AI suggests labels, but people check important ones to avoid mistakes. Using their method on forest sounds from Kenya, they saved 41% of the time needed to label sounds while keeping accuracy very high. This hybrid approach balances AI speed with human care to make sound labeling faster and reliable.
What this means in practice
- •For wildlife monitoring teams: Speed up labeling of large audio datasets by automating some annotations with human checks, saving considerable time without losing accuracy.
- •For urban sound analysts: Use a verified AI suggestion system to quickly identify speech sounds in urban recordings, reducing manual verification for certain categories.
Tested on one dataset.
Authors
Michael Barasa Murunga, Allan Vikiru
Abstract
Large-scale annotation of environmental sound recordings presents significant challenges under time and resource constraints, particularly when sound categories are imbalanced and only part of a dataset has been labeled. This paper describes the development and evaluation of a human-AI workflow used during annotation of the Kenya Forest Sounds Dataset (KFSD), using a project-stage working collection of 7,531 audio recordings across 21 categories. We investigated zero-shot classification, a feature-based ensemble, and a fine-tuned Audio Spectrogram Transformer (AST). The fine-tuned AST achieved 89.36% accuracy, 85.98% precision, 89.24% recall, and 87.58% F1 on held-out test data for binary classification of Person talking. Because this result remained below the predefined 95% threshold for autonomous annotation, model predictions were used as suggestions and subjected to human verification. Of the 5,026 initially unlabeled recordings, 2,337 were routed to a speech-verification queue and 2,689 remained for ordinary manual annotation. The verifier reviewed all 2,337 machine-labeled files and detected zero false positives, yielding 100% verification accuracy. The observed average verification time was approximately 4.2 seconds per file. At an average rate of 35 seconds per file for manual annotation, the hybrid workflow reduced work on the remaining queue from approximately 48.9 to 28.9 person-hours, saving about 20.0 hours (41%). The study documents the model-selection process, provenance-aware file organization, verification procedure, observed errors, practical limitations, and opportunities for extending AI assistance beyond one category.