OneSign unifies sign language tasks with a single adaptable model
OneSign: Unifying Sign Language Understanding Tasks with One Model
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Understanding sign language involves several tasks such as recognizing isolated signs, continuous sign sequences, and translating signs to text. Typically, each task uses a different model, making it costly and complicated to switch between them. The authors designed OneSign, a single model that handles all these tasks by adapting to the differences between visual sign input and text output. This approach uses special parts of the model dedicated to each modality while sharing common knowledge, improving efficiency and performance across tasks.
What this means in practice
- •For sign language technology developers: Build unified systems that handle multiple sign language understanding tasks without separate models for each task.
- •For multimodal ai system builders: Incorporate modality-specific expert components to better integrate visual and textual data in multimodal models.
Authors
Shiwei Gan, Yafeng Yin, Xiao Liu, Desibieer Tuerdaken, Lei Xie, Sanglu Lu
Abstract
SLU encompasses a diverse set of tasks, including ISLR, CSLR, and SLT. Although these tasks share basic semantic and linguistic foundations, they are typically addressed with task-specific architectures and training pipelines, which hinders knowledge sharing and requires costly pretraining and finetuning for each task. In this paper, we focus on two aspects of SLU tasks: (1) training and inference pipelines are highly fragmented: most methods rely on pretraining on large-scale SL datasets followed by task- or dataset-specific finetuning, which leads to multiple specialized models rather than a single checkpoint. (2) current LLM-based methods may overlook the inherent modality discrepancy between sign and text tokens, simply concatenating them and processing both modalities with the same decoder layers. In this paper, we present OneSign, a unified framework that addresses multiple SLU tasks within a single model and a single checkpoint. OneSign reformulates ISLR, CSLR, and SLT under a single training paradigm. To accommodate the heterogeneous characteristics of sign and text representations, we introduce a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture, consisting of a shared expert and modality-specific experts for sign and text tokens. A modality router dynamically activates the corresponding experts, and their outputs are aggregated to form the final token representations. By enabling modality-dependent expert specialization while preserving a shared expert path, MA-MoE can effectively model the modality differences between continuous sign representations and discrete text tokens. Extensive experiments on multiple benchmarks demonstrate that OneSign achieves competitive or state-of-the-art performance on several benchmarks, highlighting its effectiveness as a unified SLU model. Datasets are available at : https://github.com/gswycf/OneSign.