Control based method improves video motion customization without unwanted content copying
Diverse Motion Customization via Control-based Dynamic Optimization
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Motion customization in video generation often causes unwanted copying of appearance details from the example video, which spoils the new video. The authors show this happens because the model tries to copy the reference video too closely. They propose a new training approach called Control-based Motion Customization (CMC) that guides the video to have the desired motion but prevents it from copying the reference’s appearance. Their technique speeds up training and keeps the generated video diverse and faithful to the motion instructions.
What this means in practice
- •For video production teams: Create customized motion videos that accurately follow new motion instructions while preserving desired appearance from text instead of example videos.
- •For animation software developers: Build more efficient motion customization tools that reduce undesired copying of reference video features and speed up training.
Authors
Youngyoon Choi, Kihyun Kim, Jeongwoo Shin, Joonseok Lee
Abstract
Despite recent advances in video generation, motion customization remains challenging due to content leakage, where appearance attributes from the reference video unintentionally propagate into the generated output. We identify this issue as a consequence of the generative process collapsing toward the reference video, which arises from formulating the learning objective as a direct regression on the reference. To address this, we propose Control-based Motion Customization (CMC), a principled training framework that is structurally robust to content leakage. Our key idea is to steer generative dynamics toward desired motion while avoiding collapse toward the reference video, which we formalize using Stochastic Optimal Control (SOC). Under this formulation, customized videos acquire the target motion yet remain within the pre-trained model's prompt-conditional distribution, where appearance is determined by the text prompt rather than the reference video. Furthermore, to improve efficiency, we tailor the SOC formulation to motion customization by eliminating the need for an explicit reward and introducing a timestep-adaptive motion cost that focuses only on early generative stages, accelerating training by 2.5 times. Extensive experiments demonstrate that CMC effectively mitigates content leakage and achieves competitive motion fidelity while preserving the diversity of the base model across diverse scenarios.