Segue improves smooth prompt switches in streaming video generation
No Corners Cut: State-Grounded Transitions for Mid-Stream Prompt Switches in Video Generation
Computer Vision and Pattern Recognition
Summary
Making smooth changes during video generation when switching instructions is tricky because the system might skip important steps and rush to the new content. The authors present SEGUE, a way to guide video makers through these transition steps explicitly so that the video changes look natural. Their approach uses a planner that creates detailed transition prompts to bridge old and new scenes, improving quality without retraining the whole system. Tests show SEGUE improves how well videos switch between different prompts compared to previous methods.
What this means in practice
- •For video production teams: Create seamless transitions in AI-generated videos when changing scene instructions midway without artifacts or abrupt cuts.
- •For game developers: Improve dynamic video content generation in games that alters scenes or events based on player input without losing visual continuity.
Authors
Zejing Rao, Ketong Ren, Xiaoqiang Liu, Yiping Meng, Guoxin Zhang, Fan Tang
Abstract
Streaming video generators allow users to dynamically modulate video synthesis via mid-stream prompt switching. Existing streaming methods can respond to the updated instruction while still cutting corners, prematurely realizing goals or taking heuristic shortcuts that bypass necessary intermediate state changes needed for a plausible transition. In this study, we present SEGUE, a novel framework that makes this process explicit and trains the generator to execute these transitions faithfully. At each switch, a training-free planner parses the latest frame and prompts, writes a few segue prompts with roles and durations, and then hands control back to the user's prompt. Furthermore, to address the inherent difficulty of training causal models on short-lived temporal schedules without corrupting preparatory supervision, we introduce SPANDMD, which evaluates each active prompt using the full rollout as temporal context while retaining its DMD residual only within the prompt's assigned span. On OpenTrans-360, a benchmark of 1,800 switches that scores how the old state exits and the new one begins, SEGUE ranks first on all eight transition metrics and raises the overall score over the strongest baseline from 0.866 to 0.887. It also ranks first on four of six instruction-response metrics of StreamAV-Bench, while the planner transfers to frozen autoregressive generators without retraining.