AI summaryⓘ
The authors studied how to best pick samples of biomedical time-series data for human annotation, comparing random choice, a method that selects diverse samples (FAFT), and a 2D visualization-based approach (2DV). They tested these on infant movement and speech emotion tasks with experts and non-experts labeling a limited number of samples. The 2DV method generally worked best when combining labels from multiple annotators, especially capturing rare events, but it caused more variation between individual annotators, sometimes lowering model accuracy for single annotators. Random sampling was safest when annotator skill or numbers were unknown. The authors also found that 2DV made labeling more engaging for annotators. They suggest 2DV may be useful when annotation resources are not too tight.
biomedical time-seriessample selectionannotationmachine learning2D visualizationfarthest-first traversalrandom samplinginfant motility assessmentspeech emotion recognitionlabel variability
Abstract
Reliable machine-learning models in biomedical settings depend on accurate labels, yet annotating biomedical time-series data remains challenging. Algorithmic sample selection may support annotation, but evidence from studies involving real human annotators is scarce. Consequently, we compare three sample selection methods for annotation: random sampling (RND), farthest-first traversal (FAFT), and a graphical user interface-based method enabling exploration of complementary 2D visualizations (2DVs) of high-dimensional data. We evaluated the methods across four classification tasks in infant motility assessment (IMA) and speech emotion recognition (SER). Twelve annotators, categorized as experts or non-experts, performed data annotation under a limited annotation budget, and post-annotation experiments were conducted to evaluate the sampling methods. Across all classification tasks, 2DV performed best when aggregating labels across annotators. In IMA, 2DV most effectively captured rare classes, but also exhibited greater annotator-to-annotator label distribution variability resulting from the limited annotation budget, decreasing classification performance when models were trained on individual annotators' labels; in these cases, FAFT excelled. For SER, 2DV outperformed the other methods among expert annotators and matched their performance for non-experts in the individual-annotator setting. A failure risk analysis revealed that RND was the safest choice when annotator count or annotator expertise was uncertain, whereas 2DV had the highest risk due to its greater label distribution variability. Furthermore, post-experiment interviews indicated that 2DV made the annotation task more interesting and enjoyable. Overall, 2DV-based sampling appears promising for biomedical time-series data annotation, particularly when the annotation budget is not highly constrained.