Papers for

interactive media designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Game2Player scales visual agent gameplay with co-developed web games

Game2Player: Scaling Gameplay Experience for Visual Agents through Environment-Player Co-Development

Abstract: Games are a rich testbed for visual agents, demanding perception, long-horizon planning, and precise control. However, collecting human gameplay is costly, and learning from gameplay videos requires inferring missing actions. Existing datasets also cover a limited range of games, often without explicit tasks or verifiable outcomes. To address these gaps, we introduce Game2Player, a framework that co-develops game environments and players to collect diverse and verifiable gameplay experience at scale. In the Game2Player loop, a Builder agent adapts existing web games, exposes their internal state, and defines tasks with state-based progress measures and success checks. A Coding Player and a VLM Player then attempt the tasks and report problems to the Builder for repair, so that play serves as both environment validation and training data collection. With Game2Player, we build 568 verified web games with 3,436 tasks, collect 15.7K trajectories (307 hours of gameplay), and hold out 50 games as the benchmark PlayerBench. To evaluate the collected experience as training data, we train VLM agents with progress-weighted SFT, assigning greater weight to steps that advance task progress. Game2Player-9B improves progress from 14.3% to 31.2% on PlayerBench and from 30.0% to 53.1% on the external GameWorld benchmark. More broadly, our work suggests a path for visual agents to scale their gameplay experience by building, verifying, and learning from existing game worlds.

Wed 18 NovComputer Vision and Pattern Recognition
The gist
Playing video games is hard for computer programs because they need to see well, plan ahead, and control actions accurately. Collecting real human gameplay data is expensive, and watching game videos doesn't always show what actions were taken. The authors created Game2Player, a setup where a Builder program modifies games and sets tasks, and Player programs try to complete them while giving feedback to improve the games. This process gathered a large, verified collection of game-playing experiences that helped train better visual agents to play games more effectively.
Open → 2611.24154v1

SceneAgent framework makes game scene creation portable across engines

SceneAgent: A Multi-Agent Framework for Portable Game Scene Generation

Abstract: Moving a scene generator to a new game engine requires more than exporting its final output: planning, construction tools, and stored experience may also depend on the original backend. We present SceneAgent, a multi-agent framework for reusing the construction process and its experience across backends. A canonical SceneContext maintains the evolving world, while backend adapters isolate coordinate conventions, asset bindings, and export. Structured handoffs let fresh Builder sessions query this persistent state without inheriting the full construction transcript. Across generations, human-approved traces are distilled into asset-ID-free Layout skills and semantic Content skills that can be retrieved and recombined. We introduce GameSceneBench, comprising 69 indoor and outdoor prompts. Under a common asset and evaluation protocol, SceneAgent obtains human tie-adjusted preference scores of 78.2-89.4% against five baselines. Ablations show that reusable skills and the multi-agent protocol primarily improve prompt alignment and scene completeness, with additional runtime. Deployment in a second engine preserves the shared construction protocol and skill library while replacing backend integration components. Together, these results support separating persistent scene state, task-local reasoning, and reusable experience for portable scene construction.

Fri 13 NovComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Creating game scenes often involves complex tools and experience tied to one game engine, making moving those scenes to a new engine difficult. The authors introduce SceneAgent, a system that separates the scene’s state and knowledge from engine specifics, letting the same scene building process be reused on different engines. They tested SceneAgent on many indoor and outdoor scene prompts and found it better matches human preferences than other methods. This system also distills reusable skills from previous work, improving how well scenes meet their prompts and how complete they are. By isolating backend details, SceneAgent makes it easier to switch engines without losing the scene-building progress or knowledge.
Open → 2611.17821v1

TraCE improves video action understanding with explainable motion concepts

TraCE: Trajectory-Aware Concepts for Explainable Video Understanding

Abstract: Video understanding models often rely on motion, yet their explanations can obscure which motion sources support a prediction. We identify motion-source under-specification as a limitation of existing video XAI: region-centric explanations can be source-ambiguous, while motion vocabularies restricted to single-person pose sequences are source-incomplete. We propose TraCE (Trajectory-Aware Concepts for Explainable Video Understanding), a concept-based framework that discovers source-flexible motion concepts by clustering optical-flow trajectories from training videos. These concepts represent dynamic evidence from people, objects, and interactions, while object and scene concepts account for static context. The resulting typed trajectory/object/scene interface supports class prediction for action recognition and question-conditioned concept matching for video question answering (VQA). We evaluate TraCE across five random seeds on six benchmarks: NTU RGB+D Mutual, Chiral SSv2, UCF-101, KTH, TOMATO, and STAR. Relative to non-interpretable baselines, TraCE maintains competitive task performance, with an average accuracy drop of 1.0 points for action recognition and a nominal gain of 0.4 points for VQA. Motion perturbation and concept-removal tests provide complementary evidence of directional semantic alignment and decision faithfulness, while sample-level interventions illustrate targeted model debugging. Our project page is available at https://leewooil.github.io/TraCE_project_page/.

Wed 4 NovComputer Vision and Pattern Recognition
The gist
Video understanding models often use movement information but struggle to clearly show which motions influence their decisions. The authors introduce TraCE, a method that groups motion paths, called trajectories, to identify different sources of motion like people or objects. This helps explain why the model makes certain predictions in video recognition and question answering tasks. TraCE performs nearly as well as non-explainable models while offering clearer insights into its reasoning process.
Open → 2611.07009v1

Interactive narratives balance player freedom and story control with formal models

Analysing the Narrative Paradox through Formal Models and Properties in Interactive Narrative

Abstract: Interactive narratives aim to let players influence the unfolding of a story while preserving a coherent and satisfying narrative progression. This creates a central tension, commonly described as the narrative paradox, between player autonomy and narrative control. Experience Managers (EMs) manage the story by determining how the narrative progresses in response to player behaviour. Although this paradox is widely discussed, it is rarely modelled in a way that supports systematic analysis, simulation, and comparison across management strategies. This paper proposes a formal framework for modelling and analysing the narrative paradox as a trade-off between interdependent properties shaped by EM decisions. We represent interactive narrative behaviour through an extended state-transition system in which world states are connected by player actions and system events. From this model, we formalize key properties involved in the paradox such as the player agency, EM controllability, author intent, and robustness, and define quantitative indicators derived from simulated plans. These indicators make it possible to compare EM strategies according to their ability to balance freedom, control, and narrative satisfaction. A case study compares different EM policies, showing how simulations can reveal correlations and conflicts between properties and identify compromise zones in the design space. This work contributes a reusable modelling and analysis framework for studying the narrative paradox and evaluating heterogeneous interactive narratives on shared formal grounds, with a focus on supporting the comparison and testing of EM strategies.

Mon 2 NovArtificial IntelligenceHuman-Computer Interaction
The gist
Stories in interactive games face a big challenge: letting players influence the story while keeping the story clear and enjoyable. The authors created a formal way to model this challenge, called the narrative paradox, by describing how the game world changes with player actions and system events. They also developed measures to compare different ways the game manages the story, balancing player freedom and author control. They tested their approach by comparing different story management styles and showed how this helps to find good compromises between keeping the story engaging and letting players explore.
Open → 2611.02801v1

Vision language models communicate visually but differ from humans

A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs

Abstract: Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation--interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.

Tue 27 OctArtificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
The gist
Vision-language models (VLMs) can use simple pictures like red hearts to influence how they solve problems, even in surprising ways like encouraging repeated steps in coding tasks. The researchers found that while models send visual signals that seem easy for humans to understand, other models don't always interpret them the same way, creating a kind of communication gap. Sometimes these models come up with creative ways to share ideas visually with each other, but some messages are confusing to people. This work shows that VLMs have potential for visual communication but do it differently than humans do.
Open → 2610.36449v1

Matrix-game 4 offers real-time playable game worlds from video models

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Abstract: Moving from controllable videos to genuinely playable worlds requires persistent world state, executable game rules, and low-latency interaction. We outline a roadmap for the next generation of Matrix-Game that separates game execution from visual generation through an executable game harness, and introduce Matrix-Game 4.0 as the first system built on this design. A lightweight game engine maintains world state and executes interaction rules, a multimodal coding agent translates player ideas into game code and dynamic scene captions, and a renderer based on autoregressive video model transforms engine-rendered low-poly proxies into detailed visuals, conditioned on dynamic captions and memory references. This framework provides continuous guidance for visual generation and enables natural-language creation and modification of scenes and gameplay. Our renderer is trained on aligned low-poly and high-fidelity video pairs and distilled into a causal generator that generates in four-frame chunks with one or two denoising steps. Together with targeted inference optimizations, Matrix-Game 4.0 enables 480p, 60 FPS gameplay on a single NVIDIA RTX5090 with a key-press-to-display latency of 112ms. Single-step generation achieves 83FPS steady-state throughput with 18.2GiB of peak GPU memory usage. With fixed prompts, steady-state throughput also reaches 67FPS on an RTX5080 and 58FPS on an RTX5070Ti. We demonstrate customizable shooter and driving games with engine-enforced mechanics, and release code, model weights, and low-poly game assets to support research and creation of playable game worlds.

Tue 20 OctComputer Vision and Pattern RecognitionGraphics
The gist
Playing games made from controllable videos is hard because games need rules you can follow, things in the world that stick around, and fast responses to what players do. The authors show a new way of making games by keeping the game's rules and state separate from the visuals. They built a system that uses a small game engine to handle the rules, a smart language agent to turn player ideas into changes, and a video model that makes simple 3D shapes look detailed and real. This approach runs smoothly on powerful computers and can create simple playable games like shooters and driving games.
Open → 2610.27999v1

Realtime interactive video generation runs on consumer hardware

Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware

Abstract: We present Waypoint 1.5, a real-time diffusion world model for interactive video generation on consumer-grade hardware. Unlike general video diffusion models, interactive world models (iWMs) must respond to dense user controls under strict latency and throughput constraints. Waypoint 1.5 is pre-trained on 100,000 hours of diverse, control-aligned video game data across hundreds of games, and generates playable video conditioned on full keyboard and mouse input. The model includes two resolution variants that run across a wide spectrum of consumer hardware. To characterize this unique setting, we distinguish rendered FPS, latent FPS, and control rate. We describe the data pipeline, architecture, training methodology, and runtime system behind Waypoint 1.5. We evaluate interactivity through latency and throughput. Finally, we discuss the safety and ethics considerations unique to iWMs.

Tue 29 SeptComputer Vision and Pattern RecognitionHuman-Computer Interaction
The gist
Generating video that reacts instantly to a player's inputs is hard because it requires fast and accurate processing. The authors developed Waypoint 1.5, a model trained on lots of video game footage to generate playable video from keyboard and mouse commands in real time. It works smoothly on everyday computers and adapts to different hardware capabilities. They also carefully measured how quickly video frames render and respond to controls. The paper explains how to build and run such a system and considers the safety and ethical issues involved.
Open → 2609.37107v1

SAGE improves guidance stability in large generative models

SAGE: Subspace Alignment for Classifier-Free Guidance in Mixture-of-Experts Diffusion Models

Abstract: Diffusion Transformers with Mixture-of-Experts (MoE) routing are a leading recipe for scaling generative models. Classifier-Free Guidance (CFG) is essential for generation quality, yet excessively high guidance scales trigger collapse. We identify a previously unreported failure mode in their combination: the two CFG branches route independently, so their realized activations occupy different subspaces. The unconditional write then leaves the conditional subspace, and CFG amplifies that residual linearly in the guidance scale. We propose SAGE, a training-time regularizer that aligns unconditional MoE activations to the conditional subspace without restricting routing diversity, at zero inference cost. Toy experiments show that SAGE dramatically suppresses extreme drift by 9.2x. When scaled to a 1B-parameter text-to-image model, SAGE significantly improves generation quality, delivering a 9.3% boost in peak DPG-Bench performance. Extensive experiments demonstrate that SAGE consistently outperforms the baseline.

Mon 28 SeptComputer Vision and Pattern Recognition
The gist
Generating images or other content with advanced AI models can sometimes fail when the system tries too hard to follow instructions, causing it to break down. The authors found that this happens because two parts of the AI model work in different ways and create conflicting signals. They created a new method called SAGE that trains these parts to work together better without slowing down the model. This results in steadier and higher-quality outputs when making images from text.
Open → 2609.34525v1

Text to image models speed up by adapting steps to prompt complexity

Efficient Text-to-Image Generation: An Adaptive Step Schedule Controller for Diffusion Models

Abstract: Text-to-image diffusion models often use a fixed number of denoising steps, balancing time costs and image quality. However, the optimal number of steps depends on the complexity of the input text prompt. We propose an adaptive diffusion controller that dynamically adjusts the number of steps to generate high-quality images efficiently, without additional model training. By leveraging a mixture of step schedules with varying step sizes and evaluating the error term discrepancy at each timestep, our method transitions between schedules to optimize performance. Experiments on COCO and DiffusionDB show that our approach reduces inference time while maintaining visual fidelity, offering a more efficient alternative for text-to-image diffusion models.

Tue 15 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Generating images from text usually takes the same fixed amount of time, even if some descriptions are simpler and don't need as much work. The authors created a method that changes how long the computer spends on each image, based on how tricky the description is. This makes image generation faster without losing picture quality. Their approach uses different step patterns and checks for errors during the process to decide when to switch. Tests show it works well on big image sets, saving time while keeping images looking good.
Open → 2609.16572v1