Summary
Editing videos to show what would happen if objects behaved differently is hard because it requires understanding physics and predicting changes in motion. The authors propose VideoPhysEdit, a method that first reconstructs a physical simulation model of the scene, then applies edits to it and uses the simulation to create new video frames showing the resulting changes. They also provide a synthetic dataset, PCVE-RigidBench, to test how well such edits match physical outcomes. Compared to other tools, VideoPhysEdit better captures the physical effects of changes while keeping the video visually realistic. It works both on simulated and real rigid-body scenes.
What this means in practice
- •For visual effects artists: Create edited videos that realistically show how physical changes affect object movements and interactions in scenes.$Commercial implications: Enables new tools for producing realistic physical effects in video editing software sold to studios and content creators.
- •For game developers: Generate dynamic video previews of physical interactions after changing objects, aiding level design and testing without full game runs.
Authors
Conghan Yue, Yuanjie Chen, Yue Han, Ya Gao, Yunyan Xiao, WeiYao Zhang, Zhineng Chen
Abstract
Video editing has advanced substantially in recent years, with methods increasingly accounting for the visual consequences of edits, such as changes to shadows and occlusions. However, the physical consequences of edits, including changes to subsequent motion and interactions, remain less explored. We formulate this problem as physical counterfactual video editing (PCVE), which aims to generate a counterfactual video depicting the resulting motion and interactions given a source video, a physical edit, and its execution frame. PCVE is challenging because it requires understanding scene physics and inferring the downstream motion and interactions induced by a physical intervention, while paired factual and counterfactual data and dedicated evaluation metrics are lacking. We introduce VideoPhysEdit, a new training-free pipeline for PCVE in rigid-body scenes. It makes physical reasoning explicit through a novel physical scene reconstruction method that recovers a scene reproducing the observed motion and interactions under simulation, enabling the pipeline to apply physical edits as interventions and use the resulting trajectories to guide counterfactual video generation. We further construct PCVE-RigidBench, a synthetic benchmark with paired source and counterfactual target videos and physical ground truth, and introduce the Physical Edit Score. VideoPhysEdit achieves substantially higher physical edit accuracy than open-source methods and commercial models while maintaining competitive visual fidelity. Its Physical Edit Score is 0.376, the only positive score among the compared methods. Qualitative comparisons on real videos further show that VideoPhysEdit applies to real-world scenes and better depicts the downstream motion and interactions induced by the edits than the compared methods. Code: https://github.com/Hammour-steak/VideoPhysEdit