VISTA creates stylized 3D human motion from text and video style
VISTA: Video-Injected Stylized Text-to-Animation
Computer Vision and Pattern RecognitionGraphics
Summary
Generating 3D human animations that match both a description and a style can be tricky when datasets don’t have matching text, video, and motions combined. The authors introduce VISTA, a method that learns from separate text and video sources to produce motion that aligns with a text prompt while adopting the style from a reference video. It uses a special autoencoder to blend these inputs into a shared space and a diffusion model to generate animations that keep the content accurate and style expressive. Their approach lets users balance how much the animation follows the text versus the style during creation.
What this means in practice
- •For game developers: Generate stylized character animations that match narrative descriptions without needing paired style-motion-text training data.$Commercial implications: Enables game studios to produce diverse, style-rich character animations from simple text prompts and video examples, reducing animation production time.
- •For film animators: Create customized 3D animations for characters using textual descriptions combined with the style of reference videos, enhancing creative control.
Authors
Monseej Purkayastha, Anindita Ghosh, Philipp Slusallek
Abstract
We present VISTA, a two-stage framework for generating stylized 3D human motion by fusing structural content from text prompts with expressive style from reference videos, without requiring jointly paired (text, video, stylized motion) triplets. A Dual-channel Autoencoder first maps motion sequences and video clips into a shared latent manifold. A masked autoregressive diffusion backbone then operates within this manifold, injecting video-derived style through a dedicated late-fusion Dual-AdaLN pathway while preserving text-conditioned content structure. A cross-batch unpaired training protocol with latent cycle consistency enables joint learning across separate semantically rich and stylistically diverse datasets. As a proof-of-concept for controllable animation synthesis, we validate VISTA on rendered motion-capture references: it achieves the highest style recognition accuracy among video-conditioned methods while preserving competitive content alignment, and its decomposed 3-way classifier-free guidance provides independent, user-controllable calibration of the content--style balance at inference time.