SignMimic generates stable and high-quality sign language videos
SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance
Computer Vision and Pattern Recognition
Summary
Creating videos where one person perfectly imitates another’s sign language motions is challenging, especially keeping hand shapes and facial expressions consistent. The authors built SignMimic, a system that breaks down this task into parts: stabilizing overall body pose, adapting non-rigid movements like hands and face, and filling in missing details before generating the video. Their approach improves video quality, identity preservation, and smoothness across frames. It works well on multiple large sign language datasets.
What this means in practice
- •For video production teams: Create high-quality sign language mimicry videos where the signer’s identity and motion are accurately preserved for media and communication projects.
- •For sign language avatar developers: Develop more natural and stable sign language avatars by integrating motion transfer methods that keep handshape and facial details accurate over time.
Authors
Zhewen He, Junyi Yu, Haomian Huang, Zhenhua Li, Yi Fang
Abstract
We study the challenge of sign language video mimicking: given a driving video and a single reference frame, synthesize a video where the target signer reproduces the source motion while preserving identity and linguistic form. Prior pipelines entangle rigid motion, non-rigid deformation, and view-dependent completion in a monolithic generator, causing handshape drift and spatio-temporal instability. We present SignMimic, which (i) applies a TNet-based model to study SE(3) rigid canonicalization to stabilize global pose, (ii) performs non-rigid adaptation in a canonical space to preserve fine-grained articulators (hands/face) and coarticulation via NIF2D, and (iii) uses Pose-MAE-style completion before conditional video diffusion. This factorization injects geometric and linguistic priors, yielding shape and spatio-temporal consistency. On several large-scale datasets (ASL 50K, How2Sign, CSL News), SignMimic achieves state-of-the-art-level performance on video quality, identity similarity, and frame continuity while also achieving minimal loss when performing back translation (SLT) on generated videos. Ablations confirm the role of rigid canonicalization, non-rigid adaptation, and completion. Code, model checkpoints, and video examples will be released.