New position-aware metric improves handwritten math recognition scoring
PA-CDM: Position-Aware Character Detection Matching for Evaluating Handwritten Mathematical Expression Recognition
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Scoring how well a computer reads handwritten math expressions is tricky because current methods can't tell where errors happen or how serious they are. The authors created a new way to compare predictions that notices where symbols are placed, making scores better match what humans think. They also made a set of test examples with controlled changes to math expressions and showed their method agrees closely with human judgments. This new method works well without needing complex or costly tools.
What this means in practice
- •For document digitization teams: Evaluate handwritten math recognition software more accurately by detecting where reading errors occur in expressions.
- •For educational software developers: Improve feedback on handwritten math input by better matching recognition errors to human judgment of error severity.
Authors
Shiliang Luo
Abstract
Handwritten mathematical expression recognition (HMER) is conventionally scored by exact-match rates and string-similarity metrics that are blind to where an error occurs: two predictions with identical token-error counts receive identical scores whether they misplace a subscript or swap the operands of a fraction. Render-based character detection matching (CDM) aligns glyphs robustly but remains position-blind---on controlled fraction-operand swaps it scores 0.8595 where position-aware scoring yields 0.6253. Tree-edit metrics exhibit a complementary blind spot: rewrites outside the parser's normalization coverage are penalized as structural errors (0.8552 where render-based metrics score 1.0). We propose PA-CDM, a position-aware metric that couples character detection matching with position-forest encoding and divergence-level weighting; StructPerturb v2.0, a frozen benchmark of 1,340 controlled perturbation pairs across 15 type--intensity cells; and a cross-metric consistency protocol combining a sensitivity matrix, a human study, and LLM-judge calibration. In a six-annotator study, PA-CDM attains the highest correlation with human judgments among seven automatic metrics (Spearman rho=0.9535, n=990). A frontier LLM judge correlates slightly higher (rho=0.9613) but is costly, nondeterministic, and API-dependent; PA-CDM approaches it at zero marginal cost with deterministic, diagnosable behavior.