Papers for

animation and game studios

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Composing audio and text to create synchronized whole-body gestures

CFGComposer: Compositional Audio-Text-to-Motion Generation via Copula Score Correction

Abstract: Generating whole-body gestures that are simultaneously rhythmically synchronized with speech audio and semantically aligned with text descriptions is a fundamental challenge in embodied AI and social robotics. We tackle this Audio-Text-to-Motion (AT2M) problem by composing two independently-trained diffusion models at inference time and refine bias in joint inference by modeling the copula score $s_I$. A rigorous theoretical analysis reveals that naive epsilon blending corresponds to a Product-of-Experts formulation that omits the copula score $s_I$: the cross-modal interaction between audio and text that neither marginal model can capture. We propose CFGComposer: a jointly-trained AT2M diffusion model that estimates $s_I$ via a CFG analogy and injects it as an $ε$-space correction. Experiments on BEAT demonstrate best-in-class semantic fidelity (SRGR $0.9916$, FID $27.18$) with GT-level beat alignment (BA $0.1826$ vs. GT $0.1743$), and 76-83% win rate in blind pairwise human evaluation against all prior methods. Physical deployment on a Unitree G1 humanoid confirms direct sim-to-real transfer.

Wed 11 NovSoundArtificial Intelligence
The gist
Making robots or digital characters move their bodies in ways that match both what they are hearing and what they are supposed to be expressing is very hard. The authors create a new approach that combines two separate motion generators—one that listens to audio beats and one that understands text meaning—and corrects their combined output to better capture interactions between sound and meaning. Their approach produces more accurate and natural-looking gestures that align well with speech rhythm and text meaning. They tested it on a standard dataset and even showed it working on a real robot.
Open → 2611.15101v1

Hand specialized model improves accuracy in whole body pose estimation

S2HAND: Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models

Abstract: Unified whole-body pose estimators jointly predict body, face, and hand landmarks, but often degrade on small, blurred, or occluded hands. Attaching a standalone hand specialist introduces extra detection overhead, coordinate alignment, and wrist-stitching artifacts. We introduce S2HAND, a hand-specialized Sapiens2-5B framework that preserves the native 308-keypoint output interface. Specifically, S2HAND-MT adapts a lightweight 0.87M-parameter pose branch to route 40 finger heatmaps while strictly preserving the shared Transformer, auxiliary task heads, and the other 268 body, foot, and face heatmaps. S2HAND-Pose further fine-tunes three Fisher-selected Transformer blocks (267.13M parameters), trading exact preservation for stronger hand recovery: it lifts macro AUC from 57.83% to 65.73% across HInt TEST, COCO WholeBody VAL, and FreiHAND EVAL, and surpasses HInt-adapted Hand5/HRNet and stitched Hand5 systems on all five Native Pipeline benchmarks. On COCO, S2HAND-Pose improves hand AP by 12.89 points and body-plus-foot AP by 0.19 points. Across a 512-image multi-task audit, it maintains 99.60% foreground segmentation agreement with the public checkpoint, with quantified normal and pointmap drift. Its 2D coordinates also lower 3D mesh error in frozen Pose2Mesh transfer, while our optional TailGuard re-cropping module cuts S2HAND-MT's severe failure rate on NewDays from 13.45% to 10.43%. S2HAND thus resolves the whole-body hand bottleneck with either exact non-hand preservation (MT) or empirically audited trade-offs (Pose).

Sat 7 NovComputer Vision and Pattern Recognition
The gist
Whole-body pose estimation systems try to detect landmarks on the body, face, and hands all at once, but they often struggle to accurately recognize small or blurry hands. The authors developed S2HAND, which improves hand landmark detection without disrupting the rest of the pose estimation. They created two versions: one keeps the original model mostly unchanged and adds a small hand-focused part, while the other fine-tunes the model more to get better hand accuracy. Both approaches show better hand detection results across several benchmarks without hurting overall body pose accuracy.
Open → 2611.10416v1