Gamified virtual reality helps robots learn manipulation skills faster
From Gameplay to Policy: Towards Scalable Robot Data Collection via Gamified Robot-Free Interaction
Robotics
Summary
Teaching robots to handle objects well requires lots of examples, which are hard and expensive to collect in real life. The authors developed a virtual reality game called Project Kitchen that lets people practice handling objects in engaging ways without using actual robots. They designed a method called Game2Policy to turn these game actions into useful guidance for real robots, needing only a few real demonstrations to succeed. Tests showed that this approach improves robot performance and encourages diverse ways of moving objects.
What this means in practice
- •For robotics engineers: Collect diverse manipulation examples at scale without needing specific robot hardware to improve robot learning.
- •For virtual reality developers: Design engaging VR interfaces that support data collection for real-world robot training.
Authors
Zheng Li, Liang Zhu, Junzhe Wang, Huayuan Chen, Ziyun Liu, Jiahang Cao, Xinyu Sheng, Pei Qu, Yufei Jia, Ximeng Zhang, Jiarui Xie, Zizhao Yuan, Haoang Li, Yi Cai, Jinni Zhou, Jun Ma
Abstract
Learning generalizable robot manipulation policies requires large-scale and diverse interaction data, yet collecting real-world demonstrations remains costly and difficult to scale. Existing approaches to data collection are either dependent on specific robot hardware that limits crowdsourcing and transferability, or suffer from incomplete annotation and limited behavioral diversity. Inspired by how games sustain long-term human engagement, we explore an alternative paradigm that turns data collection into an engaging gameplay experience and transfers the resulting human manipulation experience to real robots. We present Project Kitchen, a VR-based gamified egocentric data collection platform that elicits diverse, goal-directed manipulation while remaining independent of specific robot embodiments and hardware, making it applicable to broader and potentially large-scale deployment. To bridge the game-to-real gap, we further introduce Game2Policy, which extracts embodiment-invariant affordance cues, including contact points and sub-goal states, from gameplay trajectories. An affordance model is pre-trained on game-collected data and then jointly fine-tuned with downstream policies using only a handful of real-robot demonstrations. Experiments show that Game2Policy improves average success rates by 10.0 points in simulation and 18.3 points on real robots in the few-shot setting. User studies and quantitative analyses further show that Project Kitchen promotes diverse manipulation behaviors and provides an engaging data collection experience. These results demonstrate the potential of gamified virtual environments as a scalable source of manipulation knowledge. The platform and code will be released upon acceptance.