Video generation improved by using keyframes with language reasoning
K2V: Generative Keyframing For Physically Plausible Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos that look realistic and follow the rules of physics is hard because current methods try to do everything at once. The authors propose a way to split this challenge by first creating important frames, called keyframes, using a language-based model that reasons about physics clearly. Then, the video generation model fills in the gaps between these keyframes, making the whole video more physically plausible. Their approach works better on special benchmarks that test if videos obey physical laws.
What this means in practice
- •For video game developers: Create game cutscenes that maintain consistent and physically believable motion across frames by controlling key states explicitly.$Commercial implications: Allows producing high-quality, physically accurate in-game videos that enhance player immersion and could be sold as content tools.
- •For film editors and vfx artists: Generate intermediate video frames that ensure physical consistency for more realistic special effects and scene transitions.
Authors
Felix O'Mahony, Daniel Olmeda Reino, Roberto Cipolla, Ayush Tewari
Abstract
Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors. This is because, in a classical video generation setting, all physical reasoning must take place simultaneously and exclusively within the diffusion process.We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physically-correct intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the Physics-IQ Verified benchmark and PhyGenBench.