Model creates 3d motion and video for any skeleton and scene
Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphicsMachine LearningRobotics
Summary
It can be hard to turn videos of characters moving into 3D animations that fit into a virtual world. The researchers developed a method that generates both realistic video and corresponding 3D motion for any character skeleton, like humans or robots, while figuring out how the camera sees the action. Their approach trains a motion model separately to learn how different skeletons move and then connects it to a video model, so the generated motion matches the visuals and scene. This lets them create, control, and place animations for diverse characters more accurately from simple text descriptions or from videos.
What this means in practice
- •For game developers: Generate and animate 3D character motions from text or video for diverse rigs to place seamlessly in interactive game scenes.$Commercial implications: Allows game studios to create customizable character animations across many skeleton types, enhancing game content creation pipelines.
- •For robotics engineers: Convert video demonstrations of robots into 3D motion models to simulate and replay robot actions in virtual environments.
Authors
Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, Yuval Atzmon
Abstract
Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.