Sign language production model adapts timing for natural motions
BLANC: Adaptive Temporal Scheduling for Sign Language Production
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating realistic sign language from text is hard because different parts of the body move at different speeds. The hands need detailed, fast movements, while the body moves more slowly. The authors designed a method that handles these differences by treating hand and body motions separately and deciding the timing dynamically. Their system, BLANC, learns when to speed up or slow down during signing to make the movements look natural and coordinated. Tests show it outperforms earlier approaches in producing sign language videos.
What this means in practice
- •For sign language app developers: Generate more natural and expressive sign language animations from text input with timing adapted to different body parts.$Commercial implications: Enables commercial sign language translation and communication apps to produce more realistic and user-friendly signing avatars.
- •For animation and game developers: Create lifelike character animations where hand and body motions have distinct timing and coordination automatically.
Authors
Youngmin Kim, Seoyeong Hwang, Sujung Hong, Seong Jae Hwang
Abstract
Sign language production (SLP) aims to synthesize expressive signing motion from written text. Modeling signing motion presents two key temporal challenges. First, fine-grained hand articulations demand higher temporal resolution than relatively slower body movements. Second, signing events unfold over variable durations, calling for adaptive temporal scheduling during generation. To address these challenges, we first introduce a Multi-Rate Vector Quantizer (MRVQ), an articulation-decoupled motion tokenizer that represents the body and both hands as separate discrete streams at distinct temporal resolutions. Building on this representation, we propose BLANC (BLock-wise Adaptive diffusioN with Articulatory Clock), a gloss-free SLP framework that generates variable-duration motion blocks causally while denoising tokens in parallel within each block. At its core, a Latent Articulatory Clock (LAC) learns temporal structure from multi-scale motion-derived partitions to predict block durations and sequence termination during generation. We further introduce inference-time body-conditioned hand refinement to improve coordination across articulators. Extensive experiments on PHOENIX-2014T and CSL-Daily demonstrate that BLANC achieves state-of-the-art performance in sign language production.