Kuration SDK helps improve virtual training data for action models
Kuration SDK: Addressing the Virtual2Real Gap via Data Curation
Machine LearningArtificial IntelligenceComputer Vision and Pattern RecognitionRobotics
Summary
Current ways to measure how well AI models learn from video games often miss important details. The authors found that popular quality metrics do not always match how well AI models actually perform in gameplay. They identify a problem called the Virtual2Real gap, where AI models look good by standard tests but behave differently in real play. To address this, they created the Kuration SDK, a toolkit that helps clean and analyze gameplay data to better prepare it before training AI models. This tool revealed why two similar models acted differently despite showing similar metrics, helping to improve future AI training.
What this means in practice
- •For game ai developers: Identify and fix data issues causing discrepancies in gameplay performance of action-conditioned AI models before training.
- •For robotics simulation engineers: Use curated physical AI datasets to reduce mismatches between simulated training and real-world robot control behavior.
Authors
Nirmit Desai, Eric Song, Mayank Sengupta, Tejal Bedmutha, Siri Reddy, Sahiti Dharmavaram, Kunal Sawarkar
Abstract
Benchmarks for measuring the quality of action-conditioned world models are still evolving and shifting away from visual similarity-based metrics to action-semantic and physically-grounded metrics. However, for domain and task-agnostic action-conditioned world model training, existing benchmarks provide a limited signal. By training and evaluating diffusion world models on CounterStrike gameplay data, we confirm that qualitative playability does not correspond with metrics such as FVD, LPIPS, and JEDi. We term this the Virtual2Real gap. We posit that, in lieu of reliable benchmarks, curating raw gameplay data and measuring a variety of diagnostic properties provides a more robust signal to bridge the gap, before the training even begins. We present several curation strategies and a general-purpose kit for physical AI data curation called Kuration SDK, which is being open-sourced with this paper. The SDK was instrumental in uncovering the root cause of the virtual2real gap in a specific case: why two world models trained on identical gameplay map, action and state distribution, behaved very differently when played in spite of having very similar LPIPS and FVD scores. Thus, Kuration SDK has the potential to uncover the root causes of Virtual2Real gap in specific datasets and accelerate development of sample-efficient training datasets.