Temporal logic improves order and timing in AI video generation
Temporal Specification Guidance for Text-to-Video Generation
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Creating videos from text prompts is hard because the timing and order of actions might be wrong. The authors introduce a technique that uses a kind of logic to clearly specify when things should happen in the video. Their method guides the video creation process without extra training, helping the video better match the requested timing and sequence. Tests show their approach works better than standard methods on many examples with different timing rules.
What this means in practice
- •For video game developers: Generate in-game cinematics that follow precise timing and action order from text descriptions without extra model training.
- •For advertising content creators: Create promotional videos that adhere strictly to scripted sequences and timing requirements specified in natural language prompts.
Authors
Avishree Khare, Hideki Okamoto, Bardh Hoxha, Georgios Fainekos, Rajeev Alur
Abstract
Text-to-video models often generate the visual concepts requested in natural language prompts but fail to satisfy their intended ordering, timing, and persistence requirements. We propose representing these requirements as specifications in a formal Temporal Logic, and introduce Temporal Specification Guidance (TSG), a training-free inference-time framework that turns temporal specifications into executable objectives for guiding video generation during sampling. TSG scores traces of visual concept evidence using differentiable quantitative semantics and uses the score's gradients to construct a specification-guided denoising direction. We instantiate this framework with video-text attention traces, efficient differentiable scores for Metric Temporal Logic, and gradient-derived attention biases. This instantiation requires neither an external concept classifier nor backpropagation through the video transformer. Its attention-steered forward pass yields a denoising direction that augments classifier-free guidance (CFG). On a benchmark of 220 prompts spanning 11 categories of temporal constraints, including ordered object entry and state transitions, this instantiation improves specification satisfaction accuracy over CFG by 9.5-20 percentage points across five models, with minimal memory overhead. These results demonstrate the potential of temporal specifications and differentiable satisfaction objectives as a flexible interface for controlling video generation.