Motionshift adapts motion to fit scenes for realistic video transfer
MotionShift: Environment-Compatible Motion Transfer via Planner-Evaluator Synergistic Adaptation
Computer Vision and Pattern Recognition
Summary
Many motion transfer methods copy movements from one video to another but fail to make sure the movements make sense for the new scene, like obeying physical rules. The authors introduce MotionShift, which uses a language model to plan motions based on the scene's meaning and a vision-language model to check and fix these motions using visual cues. This back-and-forth makes sure the motion fits well in the new environment without needing extra training. Finally, the adapted motion guides a video generator to create realistic videos in different scenes.
What this means in practice
- •For film and game developers: Create realistic character movements automatically adapted to diverse environments for movies and games using existing video sources.$Commercial implications: Enables production studios to generate scene-appropriate animations without manual corrections, improving efficiency and realism.
- •For augmented reality app developers: Generate motions for virtual characters that align physically and visually with real-world scenes captured by users.
Authors
Yanchen Liu, Junyao Gao, Yanan Sun, Wensheng Chen, Guangming Lu, Wenjie Pei
Abstract
Existing motion transfer methods excel at exact, rigid motion replication from reference videos to target scenes, whereas they often fail to account for the compatibility between the transferred motion and the target scene environment in accordance with physical laws. In this work, we propose MotionShift, a novel motion transfer framework that achieves high-fidelity motion transfer while ensuring physical compatibility with the target scene. In particular, we design a planner--evaluator iterative motion adaptation mechanism, in which a Large Language Model (LLM) acts as the planner to generate motion trajectories by leveraging its strong semantic reasoning capabilities, while a Vision-Language Model (VLM) serves as the evaluator to rectify the motion based on its superior visual perception and scene understanding. Through this planner--evaluator synergistic reasoning process, the motion is adapted to the target scene in a training-free manner. The adapted motion is then used as a generative condition for an image-to-video (I2V) model to render videos in the target scene. Extensive experiments demonstrate that MotionShift achieves high-fidelity and physically compatible motion transfer across diverse and complex scenes. Moreover, we show that MotionShift can be readily integrated into existing trajectory-conditioned I2V models, endowing them with realistic motion transfer capabilities.