Dancer pose improves tagging of Greek traditional music alongside audio
Pose-Aware Multimodal Automatic Tagging for Greek Traditional Music
SoundComputer Vision and Pattern Recognition
Summary
Automatic tagging of music often uses just the sound, but in performances like Greek traditional music, dancing and visual cues also carry important information. The authors studied whether tracking dancers’ body poses from video can help computers tag music better. They created a method to extract dancers’ skeleton movements from live recordings and combined this with audio features. Even though audio alone is strong, adding pose data helped the system recognize music details more accurately.
What this means in practice
- •For music streaming platforms: Enhance music tagging accuracy for niche genres by integrating dancer pose information alongside audio features.$Commercial implications: This enables richer metadata for Greek traditional music, improving search and recommendation features for targeted audiences.
- •For cultural heritage institutions: Use pose-based tagging to better catalog and preserve traditional music performances with multimodal descriptors.
Tested on one dataset.
Authors
Alexandros Alexiou, Charilaos Papaioannou, Alexandros Potamianos
Abstract
Automatic tagging is a core task in Music Information Retrieval (MIR), yet most tagging systems exploit only audio. Live music performance is inherently multimodal, as semantic labels such as instruments, regional styles, and dance forms are encoded simultaneously across acoustic, visual, and embodied performance cues. This is especially true of culturally specific repertoires such as Greek traditional music, which remain underrepresented in MIR benchmarks. In this paper, we investigate whether the use of dancer pose provides complementary information for automatic tagging in Greek traditional music beyond audio. Using the Lyra dataset, we extend prior audio-only work by extracting aligned video features and pose-derived skeleton streams, enabling an experimental setting for multimodal auto-tagging. We further introduce an automated pipeline for extracting primary-dancer skeleton sequences from in-the-wild dance footage, combining dance-scene detection, multi-person tracking, dancer selection, pose estimation, and quality filtering. We compare unimodal, all bimodal combinations, and trimodal systems using multiple fusion strategies. Audio remains the strongest single modality (AST: macro ROC-AUC 0.821), while skeletons, though weak in isolation, enhance performance through multimodal fusion. The best trimodal system improves macro ROC-AUC by about 4 percentage points over the strongest audio baseline.