Signrag improves gloss free sign language translation with retrieval enhanced large models
SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation
Computation and LanguageComputer Vision and Pattern RecognitionMultimedia
Summary
Translating sign language directly into spoken language without using sign glosses (written labels for signs) is hard, especially for popular language models designed in a different way. The authors of this paper designed a new system called SignRAG that helps large language models better understand sign videos and find useful examples from a database to improve translations. Their method combines special training steps and reinforcement learning to make sure the retrieved examples help, not confuse, the translation. SignRAG beats previous methods on public sign language benchmarks, including outperforming models that used glosses on some tests.
What this means in practice
- •For sign language technology developers: Build sign language translation apps that directly convert sign video to text without gloss annotations using enhanced retrieval-based language models.$Commercial implications: Enables more accurate and scalable sign language translation products for communication and accessibility by reducing reliance on gloss labeling.
- •For multimodal ai researchers: Create multimodal models that better align visual and language data by integrating hierarchical training and retrieval-aware fine-tuning.
Authors
Zhi Rao, Yucheng Zhou, Qianran Sun, Yiqing Huang, Longcan Yuan, Jiayi Hou, Chengwen Yao, Lin Cheng, Donghui Sun, Xiaoxin Chen, Jun Wan
Abstract
Contemporary decoder-only large language models (LLMs) have demonstrated strong capabilities across a wide range of domains. However, existing pretraining paradigms for gloss-free sign language translation (SLT) are largely designed around conventional encoder-decoder pretrained language models, which limits their direct applicability to decoder-only LLMs. To address this limitation, we propose SignRAG, a unified framework combining hierarchical pretraining, target-domain retrieval augmentation, and retrieval-aware reinforcement fine-tuning. Hierarchical pretraining first learns linguistically grounded sign representations and then jointly aligns the sign encoder with an LLM, mitigating cross-modal optimization imbalance. For downstream adaptation, SignRAG complements parameter-based fine-tuning with a target-domain retrieval gallery that provides instance-specific translation cues. To ensure that retrieved contexts are used appropriately, we further introduce Retrieval Utility-Guided Reinforcement Fine-Tuning (RUG-RFT), which combines translation-quality and retrieval-utility rewards to encourage beneficial retrieval use while suppressing harmful reliance. Experiments on multiple SLT benchmarks establish new state-of-the-art performance. In particular, to the best of our knowledge, SignRAG is the first gloss-free approach to outperform gloss-supervised methods across all reported metrics on CSL-Daily. Our code has been released at \href{https://github.com/shahelaojieraozhi/SignRAG}{GitHub}, together with models of different sizes to support future academic research.