SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors explore how to improve large language models (LLMs) for translating sign language videos directly into text without using glosses (intermediate representations). They identify two main problems: LLMs struggle to understand visual inputs because they are trained mostly on text, and current methods make models focus too much on text instead of visual cues. To fix this, they create a training process that better aligns visual features with text and teach the model to rely more on visual information through a special distillation technique. Their approach, called SignLlama, performs well on several sign language translation datasets without needing extra data.
Large Language ModelsGloss-Free Sign Language TranslationVisual FeaturesText FeaturesPseudo-GlossCTC PretrainingVisual-Prioritized DistillationAutoregressive FrameworkSign LanguageRepresentation Learning
Authors
Shiwei Gan, Xiao Liu, Yafeng Yin, Zhiwei Jiang, Bowen Guo, Lie Xie, Sanglu Lu, Hongkai Wen
Abstract
Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss-Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which leverages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the second issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining.