SignRR: Retrieve and Refine Real Motion for Sign Language Production
2026-08-28 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of generating realistic sign language motions from spoken language. They note that existing methods either create motions from scratch, which can miss rare hand shapes, or stitch together real motion segments, which can cause awkward transitions. Their approach, called SignRR, starts with real motion pieces and then fine-tunes them to create smooth, consistent signing sequences. Experiments show their method improves the translation quality from sign language to text while keeping good motion realism.
Sign language productiongloss-to-pose generationgenerative modelsretrieval-based methodsco-articulationResidual VQ-VAElatent spaceback-translationPHOENIX14T datasetCSL-Daily dataset
Authors
Fidel Omar Tito Cruz, Angie Sanchez Marquina, Summy Farfan, Gissella Bejarano
Abstract
Sign language production (SLP) aims to generate continuous signing motion from spoken language, often through gloss-to-pose generation. Prior work mainly follows two paradigms. Generative models synthesize motion from a learned prior or from noise, without reference to an observed signing instance, making rare hand configurations and signer-specific articulation difficult to preserve. Retrieval-based methods reuse real, well-articulated motion segments, but concatenating segments from different signers and co-articulation contexts can introduce rhythm and style inconsistencies across the full sequence, not only at segment boundaries. These limitations suggest a complementary solution: use retrieval to provide realistic articulation, and use learned refinement to impose the global coherence that retrieval alone lacks. We therefore propose retrieve-and-refine, a paradigm that starts from real retrieved motion and refines it into a globally coherent signing sequence rather than generating motion from scratch. Our framework, SignRR, initializes motion from a dictionary of real sign segments and refines the full sequence with a part-aware Residual VQ-VAE, where residual quantization preserves fine hand articulation and temporal length differences are handled in the latent space. Experiments on PHOENIX14T and CSL-Daily show that SignRR achieves state-of-the-art back-translation performance while maintaining competitive pose quality.