TouchéWorld combines camera and hand controls for interactive video worlds

TouchéWorld tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Exploring virtual worlds by moving a camera around is common, but adding hand motion lets people interact more naturally with objects and characters. The authors built TouchéWorld, a system that makes videos showing both camera movement and hand gestures together in a way that matches user controls over time. They created training data by aligning and cleaning real first-person videos and adapted a video generation model to include detailed hand poses and movements. This lets users explore and interact with generated scenes in real time, seeing immediate visual feedback based on their hand and camera actions. They also made TouchéBench, a set of tests to check how well the system follows hand motions and coordinates them with camera views.

What this means in practice

  • For virtual reality developers: Create immersive virtual environments where users’ camera movements and hand gestures control interactions in real time with generated objects and characters.$Commercial implications: Enables VR experiences that combine realistic visual exploration and natural hand-driven interactions, supporting commercial VR products and games.
  • For human-computer interaction engineers: Develop interactive systems that integrate continuous camera and detailed hand motion inputs to improve online user control and feedback in virtual environments.

Authors

Jiaao Yu, Yanjian Zhang, Haoyang Jiang, Guillaume Wisniewski, Nadi Tomeh, Shijie Jia

Abstract

Recent video world models allow users to explore generated environments through camera navigation. Continuous hand motion provides a fine-grained action channel for interacting with the people and objects encountered during exploration. We present TouchéWorld, a framework that integrates gesture-driven interaction into a camera-controlled video world model. Given an initial image and a scene-description prompt, the model generates video conditioned on temporally aligned camera and hand controls. We construct joint camera--hand training data from heterogeneous egocentric videos through annotation completion, consistency-based filtering, and temporal alignment. Building on a pretrained camera-controlled world model, we combine spatial hand-skeleton conditioning with numerical hand-motion features through parameter-efficient adaptation. Causal adaptation and few-step distillation support chunk-level online generation, allowing users to update their controls based on generated visual feedback. We further introduce TouchéBench to evaluate hand-motion following, camera--hand coordination, online interaction, and object response. Together, these components establish a framework for studying continuous exploration and hand-driven interaction within generative worlds.
View PDFOpen arXiv