Kinematic capture system tracks both hands with one 360 camera
Kinematic Interface for the Wild: Modular Bimanual Loco-Manipulation Capture from 360$^{\circ}$ Cameras Alone
Robotics
Summary
Capturing precise motion of both hands while manipulating objects is tricky because cameras often struggle to see both hands and track their position accurately. The authors present a system called KIWI that uses a single 360-degree camera to solve this by splitting the tracking tasks between the front and back lenses. The back lens maps the whole room and localizes the hands and even a head camera, while the front lens focuses on recording the hand movements. They show this system can very reliably track both hands with minimal errors and can reconstruct the scene in 3D from just the wrist camera views.
What this means in practice
- •For robotics engineers: Capture accurate bimanual manipulation data for robot learning and control without needing extra specialized tracking hardware.
- •For virtual reality developers: Track users’ hands and manipulation activities within a room-scale VR setting using a single wearable 360 camera.
Authors
Benjamin Yang, Weiying Wang, Shenggao Li, Keming Yan, Sasha Wilkinson, Zelin Wang, Yip Fun Yeung, Lingfeng Sun
Abstract
A wrist-mounted camera for UMI-style data collection must do two jobs: record the manipulation and localize in the scene. Most handheld devices localize online from workspace-facing views crowded by hands and objects, or add dedicated tracking hardware. Room-scale bimanual capture therefore still tends to instrument the operator or the scene for accurate localization. We present KIWI (Kinematic Interface for the Wild), a capture kit whose only electronics are off-the-shelf cameras. Our core system splits the two jobs across the two lenses of a 360-degree camera. The rear lens faces the room and builds a shared metric map that registers both hands, and an optional head camera, in one frame without workspace co-visibility; the front lens records the manipulation, and offline IMU fusion bridges front-lens tracking loss. Through our quick-release plate, the camera module attaches to chopstick grippers, parallel-jaw grippers, hand-wrist mounts, or robot flanges. Across six bimanual recordings, combining the rear and front lenses failed to localize only 0.1% of query frames, whereas front-only bimanual feature alignment failed on 24.8% of frames and lost one recording entirely; against evaluation fiducials, localization error stayed within 4.5 mm. KIWI's recovered poses were sufficiently consistent for the four wrist streams alone to reconstruct the scene as a 3D Gaussian splat.