Human and object 4d reconstruction improves with interaction aware re rendering
HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering
Computer Vision and Pattern Recognition
Summary
Capturing how people interact with objects in videos from just one camera is tricky because cameras only see one side and objects can block each other. The authors created a system, HORIA, that breaks down a video into clear steps of interaction, describing each with short text. Then, it uses these texts to create better views of the scene that fill in the missing parts, making the 3D reconstructions of humans and objects more complete and accurate over time.
What this means in practice
- •For augmented reality developers: Create interactive AR experiences that accurately track and display humans manipulating objects from a single camera feed despite occlusions.
- •For robotics engineers: Improve robots’ ability to understand and predict human-object interactions using detailed 4D reconstructions from monocular video input.
Authors
Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung, Kyoung Mu Lee
Abstract
Joint 4D reconstruction of a human and object from a monocular video is essential for understanding how people interact with their environment. While recent methods have made noticeable progress, capturing such human-object interactions remains difficult due to two main challenges. First, a video typically contains composite semantic context spanning multiple interaction phases, making it difficult to track the temporal changes of the interaction in the reconstruction. Second, under human-object occlusion, a single-view observation contains incomplete visual information about the interaction, resulting in geometrically inconsistent reconstructions. To address these challenges, we introduce HORIA, built upon two core designs. First, we decompose the entire input video into temporally grounded and semantically distinct interaction phases with motion-aware captioning, obtaining a specific text description for each phase. Second, conditioned on these text descriptions of the interaction, we propose an interaction-aware re-rendering that compensates for the limited single-view observation. Specifically, we render the initially reconstructed 4D human and object from novel viewpoints and refine the renderings with a text-to-video diffusion prior, which provides auxiliary novel-view supervision beyond single-view cues. As a result, our framework produces temporally coherent and accurate 4D reconstructions under diverse interaction scenarios, achieving state-of-the-art performance.