Casual videos reveal how hands move objects with joints
Track, Articulate, Act: Generating Articulation from Casual Human Videos
Computer Vision and Pattern Recognition
Summary
Figuring out how things like doors and drawers move from just watching regular videos is tricky because these objects have parts that move in different ways. The authors created a method that looks at simple videos from one camera to find how these parts move and how hands push or pull them. They use smart tools that already know how to guess 3D shapes and motion, then connect that information to understand joints and movement patterns. This helps turn everyday videos into 3D models that robots could use to learn how to interact with these objects.
What this means in practice
- •For robotics engineers: Generate 3D models of movable objects and corresponding hand motion from casual videos to simulate and train robot manipulation tasks.
- •For automation system developers: Create simulation-ready articulated object assets from everyday videos without special sensors or manual input to speed up design of interactive systems.
Authors
Jiaming Zhang, Homanga Bharadhwaj
Abstract
Human videos contain rich causal evidence for robot manipulation: they reveal how hand motion induces object motion and produces task-relevant changes in object state. In this work, we study articulated objects such as doors, drawers, cabinets, laptops, ovens, and hinged containers that are ubiquitous in daily life and present unique challenges for embodied interaction. These objects cannot be represented by a single pose; their motion depends on the underlying parts and joints. We introduce a real-to-sim framework that reconstructs a simulation-ready articulated object and hand-object interaction from a casual monocular RGB video, without RGB-D or multi-view input, prior scans, manually specified joints, or robot demonstrations. Our key insight is that dense 3D point tracks provide an embodiment-agnostic articulation cue: points on the fixed link remain approximately stationary, while points on the moving link follow coherent revolute or prismatic motion. Our method segments the links, estimates the joint and its state trajectory, reconstructs an articulated asset, and aligns the recovered 3D hand motion with the object. Central to our approach is a modular recipe that repurposes powerful pretrained models for single-image 3D reconstruction, mesh segmentation, and 3D scene flow, connecting their predictions through explicit geometric reasoning to infer articulation. We use the reconstructed articulated object and the human hand trajectory to replay interactions through contact in MuJoCo. The framework shows how pretrained vision models and explicit motion reasoning can turn casual human videos into articulated object models suitable for downstream embodied interactions. https://track-articulate-act.github.io/