Isekai3D SDG creates diverse synthetic 3D gameplay visual data
Isekai3D SDG: A Platform for Diverse Synthetic Gameplay Visual Data
Computer Vision and Pattern Recognition
Summary
Collecting lots of useful video and image data from 3D games is hard and expensive, especially when you want detailed info about the game's world and characters. Isekai3D SDG is a tool that helps by simulating gameplay and capturing many types of visual data along with exact details about the game state. It does this in two steps—first running the game simulation, then replaying it to produce different kinds of images or videos without repeating the gameplay. This makes gathering rich, varied training data easier and faster for computer vision and game AI researchers.
What this means in practice
- •For game developers: Generate diverse synthetic gameplay visuals for training AI agents without rerunning expensive simulations.
- •For computer vision engineers: Produce richly annotated synthetic video and image data to develop and test vision algorithms in virtual worlds.
Authors
Rafal Karp, Gene Liu, Shih-Hsin Li, Eric Courtemanche, Shengze Wang, Josef Spjut, David Pankratz, Mischa Kolbe, Michael Stengel, Sven Elflein, Adam Grygielski, Liwen Wu, Anjul Patney
Abstract
We present Isekai3D Synthetic Data Generation (SDG), a platform for simulating and capturing synthetic 3D gameplay for research in vision and world models, neural graphics, gameplay agents, and other applications. The platform combines reusable environment and character assets with Unreal Engine plugins in a two-stage workflow. First, world simulation generates gameplay while recording the world state, character movements, actions, and camera tracks needed for replay. Second, the recorded simulation is replayed to capture selected outputs, such as rendered video, images, and depth maps, at a specified resolution and frame rate under selected rendering and camera configurations. Separating simulation from capture allows the same recorded gameplay to produce multiple output variants without repeating the original simulation. Visual outputs are accompanied by frame-aligned camera, character, and action metadata. The platform uploads generated data directly in a training-friendly format, reducing intermediate storage, preparation time, and manual processing. This report describes the design of our overall system, including the platform's assets, plugins, and their functionality, explaining how they work together to generate diverse synthetic gameplay data.