Reconstructing Humans and Objects in Interaction using Large Reconstruction Models
2026-08-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a method called MILO to better estimate how people interact with objects in 3D using just one image. Instead of traditional techniques that rely on fitting models directly to 2D images, MILO uses big reconstruction models to create a detailed 3D mesh showing both humans and objects together. Then, they separate the mesh into human and object parts to accurately model the interaction. Their approach simplifies the process and performs better than older methods on various tests.
3D human-object interactionLarge Reconstruction Modelsparametric body model3D mesh segmentationsingle image reconstructionobject alignmentdepth ambiguityocclusion
Authors
Agniv Chatterjee, Georgios Pavlakos
Abstract
Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.