Sign language translation accuracy drops when tested with new signers
Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation
Computation and LanguageArtificial Intelligence
Summary
Sign language translation models seem to work well when tested on people they have already seen, but the authors found that their performance drops a lot when tested on new signers they haven't seen before. This means current tests might overestimate how good these models really are at understanding sign language from different people. The authors suggest new ways to test and organize data so we get a clearer picture of how well these models truly work for everyone.
What this means in practice
- •For sign language technology developers: Evaluate sign language translation models more reliably by using signer-independent testing to ensure models generalize to unseen users.
- •For dataset curators: Create sign language datasets with explicit signer-independent and sentence-disjoint splits to enable fair and consistent benchmarking of translation models.
Authors
Keren Artiaga, Sabyasachi Kamila, Haithem Afli, Conor Lynch, Mohammed Hasanuzzaman
Abstract
Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev/test. This raises concerns about whether models truly generalise or instead rely on signer-specific regularities. We conduct signer-fold cross-validation on GFSLT-VLP, GASLT, and SignCL, three leading, publicly available, gloss-free SLT models, on CSL-Daily and PHOENIX14T. Under signer-independent evaluation, performance drops sharply: on PHOENIX14T, GFSLT-VLP falls from BLEU-4 21.44 to 3.59 and ROUGE-L 42.49 to 11.89; GASLT from 15.74 to 8.26; and SignCL from 22.74 to 3.66. We also observe that in CSL-Daily many target sentences are performed by multiple signers, so common splits can place identical sentences in both training and test, inflating absolute scores by rewarding recall of recurring sentences rather than genuine generalisation. These findings indicate that signer-dependent evaluation can substantially overestimate SLT capability. We recommend: (1) adopting signer-independent protocols to ensure generalisation to unseen signers; (2) restructuring datasets to include explicit signer-independent, sentence-disjoint splits for consistent benchmarking; and (3) reporting both signer-dependent and signer-independent results together with train-test sentence overlap to improve transparency and comparability.