Graph neural networks improve handwritten music note placement accuracy
Data-Driven Glyph Positioning for Handwritten Music Synthesis
Computer Vision and Pattern Recognition
Summary
Recognizing handwritten music notes from scanned images is hard because there isn’t enough labeled data. The authors studied how synthetic handwritten music scores are made by placing note parts (glyphs) with fixed rules, which do not match real handwriting well. They found that using data-driven positioning reduces mistakes, and training a model called a graph neural network to predict better placements reduces errors even more. This improved synthetic data helps train detectors to find music notes in real handwritten scores more accurately.
What this means in practice
- •For music technology developers: Generate more realistic synthetic handwritten music for training recognition software, improving detection of handwritten music notation.
- •For document analysis engineers: Use improved glyph positioning techniques to enhance training data quality for detecting handwriting elements in other complex notated documents.
Authors
Elona Shatri, Jiří Mayer, Jan Hajič
Abstract
Synthesised data is a common strategy to address the lack of annotated handwritten music scores in Optical Music Recognition (OMR). Smashcima is a recent synthesis framework that assembles pages of handwritten scores by sampling glyph images from MUSCIMA++ and positioning them with hard-coded rules. In this work we ask whether these rules are grounded empirically and whether a learned model for context-dependent positioning can further improve synthesis quality. We extract inter-glyph distributions from around 285 thousand annotated relationships across MUSCIMA++ and OmniOMR, finding that several key parameters deviate systematically from real handwriting. Replacing these constants with empirical means reduces positioning RMSE by up to 60%. We then train a Graph Neural Network (GNN) on the Music Notation Graph structure to predict context-dependent offsets, achieving an additional 50 to 85% RMSE reduction on stems and beams by modelling how neighbouring glyphs constrain placement. Cross-dataset analysis shows that spacing differences between the two corpora are explained by note density (3.3), not style, and that duration-proportional spacing does not hold in handwriting (R2 = 0.001). In downstream evaluation, a YOLOv8 detector pretrained on GNN-positioned synthetic data achieves +30% mAP50 over the baseline and +11% over empirical means when fine-tuned on 200 real handwritten staves, showing that positioning accuracy matters for downstream OMR.