Monocular 3D human localization improved by visual features and social context
MoSoLoco: Monocular 3D Human Localization through Visual Latent Features and Social Context
Computer Vision and Pattern RecognitionRobotics
Summary
Figuring out where people are in 3D space using just one camera is useful for robots moving around. The authors created MoSoLoco, a method that uses both the detected 2D body points and the internal visual clues from the pose detector to better estimate where people stand. It can also consider nearby people in the same image to improve accuracy. Tests on real-world datasets show that this approach is more accurate than older methods and can run efficiently on robot hardware.
What this means in practice
- •For robotics engineers: Locate multiple people in 3D using a single camera with improved accuracy and small computational footprint for robot navigation and motion prediction.
- •For autonomous vehicle developers: Enhance pedestrian position estimation in crowded scenes from monocular cameras to support safer trajectory forecasting and vehicle planning.
Authors
Yasamin Borhani, Javad Khoramdel, Yang Gao, Po-Chien Luan, Alexandre Alahi
Abstract
Estimating the metric 3D position of people from a monocular camera is important for robot navigation and motion prediction. Lightweight pose-based localizers are attractive for robotic hardware with limited computational resources because they operate on compact 2D keypoints. However, they typically discard the richer visual representations computed by the pose estimator. We introduce MoSoLoco, a per-frame monocular 3D human localizer that combines the pose detector's internal visual features with the detected 2D skeleton, without requiring an additional RGB or depth encoder. MoSoLoco can further incorporate sparse same-frame multi-person context to refine individual position estimates. Experiments on JRDB and NuScenes show that reusing pose-detector representations improves localization, reducing the best JRDB error by approximately 23 percent relative to the strongest evaluated keypoint-based baseline. A sparse social-interaction module incorporates nearby detections from the same image for further refinement. Controlled trajectory-forecasting experiments show downstream benefits, while the complete pipeline remains suitable for efficient Jetson AGX Xavier inference.