Realtime interactive video generation runs on consumer hardware
Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Generating video that reacts instantly to a player's inputs is hard because it requires fast and accurate processing. The authors developed Waypoint 1.5, a model trained on lots of video game footage to generate playable video from keyboard and mouse commands in real time. It works smoothly on everyday computers and adapts to different hardware capabilities. They also carefully measured how quickly video frames render and respond to controls. The paper explains how to build and run such a system and considers the safety and ethical issues involved.
What this means in practice
- •For video game developers: Create more responsive and playable AI-generated game video that reacts instantly to player inputs on typical consumer devices.$Commercial implications: Allows development of AI-driven games playable on regular computers with real-time responsiveness uncommon in current offerings.
- •For interactive media designers: Build interactive video experiences that respond immediately to user controls without high-end hardware.
Authors
Rajit Rajpal, Shahbuland Matiana, Liew Wei Pyn, Anmol Agarwal, Ryan Craig, Andrew Lapp, Mithun Hunsur, Sami BuGhanem, Scottie Fox, Aaron Sanders Carson Poole, Irene Park, Dave Rossi, Spencer Frazier, Louis Castricato
Abstract
We present Waypoint 1.5, a real-time diffusion world model for interactive video generation on consumer-grade hardware. Unlike general video diffusion models, interactive world models (iWMs) must respond to dense user controls under strict latency and throughput constraints. Waypoint 1.5 is pre-trained on 100,000 hours of diverse, control-aligned video game data across hundreds of games, and generates playable video conditioned on full keyboard and mouse input. The model includes two resolution variants that run across a wide spectrum of consumer hardware. To characterize this unique setting, we distinguish rendered FPS, latent FPS, and control rate. We describe the data pipeline, architecture, training methodology, and runtime system behind Waypoint 1.5. We evaluate interactivity through latency and throughput. Finally, we discuss the safety and ethics considerations unique to iWMs.