SignRefine improves sign language video clarity using keypoint data
SignRefine: Adapting Foundational Video Models for Sign Language Generation
Computer Vision and Pattern Recognition
Summary
Making videos of sign language that show clear hand and facial movements is very hard. Most video-generating tools are trained on regular videos with speech, so they create confusing signs with mistakes. The authors developed SignRefine, a new model that starts with an existing video tool and then focuses on fixing hands and faces based on simple movement signals. They also created a big collection of sign language videos to train their system. SignRefine produces clearer and more understandable signing videos that people who use sign language prefer.
What this means in practice
- •For video content creators: Generate clear sign language videos from simple motion data to improve accessibility in media.
- •For human computer interaction engineers: Create more accurate sign language avatars that respond to 2D movement inputs for communication tools.
Authors
Anton Pelykh, Edward Fish, Ozge Mercanoglu Sincan, Richard Bowden
Abstract
Sign language video generation demands precise hand and facial articulation, yet modern video diffusion models, trained predominantly on spoken-language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model's prior toward accurate articulation. To enable this work and support broader sign language research, we present NVSign, a large-scale dataset of video content natively produced in sign language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign language users for visual quality and comprehensibility in more than 80% of comparisons.