Papers for

film editors and vfx artists

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Video generation improved by using keyframes with language reasoning

K2V: Generative Keyframing For Physically Plausible Video Generation

Abstract: Video diffusion models are able to generate samples with high-quality appearance and textural qualities. However, content produced by these methods is known to exhibit significant physical errors. This is because, in a classical video generation setting, all physical reasoning must take place simultaneously and exclusively within the diffusion process.We propose to resolve these issues by redistributing the task of physical reasoning onto a language model-backed keyframe generation module, which allows for explicit reasoning in language, and fine-grained control over the appearance of the video at important moments in the video. We then condition a video generation model with these keyframes, reducing the role of the diffusion model to the simpler task of interpolating between physically-correct intermediate states, rather than inferring an entire sequence with diffusion. Results demonstrate that our approach significantly outperforms existing approaches, achieving state-of-the-art performance on the Physics-IQ Verified benchmark and PhyGenBench.

Thu 12 NovComputer Vision and Pattern Recognition
The gist
Generating videos that look realistic and follow the rules of physics is hard because current methods try to do everything at once. The authors propose a way to split this challenge by first creating important frames, called keyframes, using a language-based model that reasons about physics clearly. Then, the video generation model fills in the gaps between these keyframes, making the whole video more physically plausible. Their approach works better on special benchmarks that test if videos obey physical laws.
Open → 2611.17324v1