Sign language production improved by natural motion paths on joint manifolds
Naturalness-guided Manifold Flow Matching for Sign Language Production
Artificial Intelligence
Summary
Generating sign language from text is hard because human body movements follow complex joint rotations, not simple straight lines. The authors point out that earlier methods used straight-line paths that don’t match how joints actually move. They propose a new approach that follows the true curved space of joint rotations, making sign motions look more natural and accurate. Their method combines mathematical paths with learned adjustments guided by how natural the motions appear.
What this means in practice
- •For sign language app developers: Create more realistic and natural sign language animations from text inputs for communication apps using the new manifold-based motion generation technique.$Commercial implications: Enables selling enhanced sign language video generation features to accessibility app companies by improving animation quality and naturalness.
- •For animation software engineers: Incorporate motion interpolation methods that respect joint rotation geometry to improve character animation realism in 3D software tools.
Authors
Jiayi He, Shengeng Tang, Sisi You, Yanbin Hao, Lechao Cheng, Richang Hong
Abstract
Sign Language Production (SLP) aims to generate sign motions from text. Conditional Flow Matching methods have achieved strong performance in SLP by constructing conditional paths that transform a source distribution into a target distribution. However, existing methods construct these paths via linear interpolation, whereas the rotational geometry of human joints confines valid joint rotations to a manifold embedded in Euclidean space. Consequently, linear interpolation between two sign motions leaves this manifold and ignores the motion distribution on it. In this paper, we revisit SLP from the perspective of manifold transport and propose a Naturalness-guided Manifold Flow Matching framework, termed \textbf{SignNMFlow}, which constructs conditional paths directly on the motion manifold by jointly considering geometric efficiency and the motion distribution. Specifically, we exploit the intrinsic geometry of the manifold and introduce a motion naturalness measure to characterize the motion distribution. By minimizing the kinetic energy under this measure, we learn a naturalness-guided interpolation that couples a closed-form geodesic, which provides geometrically efficient transport, with a learnable deviation that incorporates the motion distribution, thereby significantly improving the fidelity of generated sign motions. Extensive qualitative and quantitative evaluations demonstrate the effectiveness of this work.