Robotic benchmark reveals challenges in mobile manipulation tasks
RMMBench: A Comprehensive Benchmark for Robotic Mobile Manipulation
Robotics
Summary
Robots are getting better at understanding language and the world around them, but it is hard to test how well they can do complex tasks that involve moving and handling objects. The authors created RMMBench, a set of 70 tasks that make robots follow instructions to navigate and manipulate things over time. Their tests show that current robot models still struggle with knowing exactly where they are and manipulating objects accurately in the real world. This benchmark helps highlight what needs to improve for robots to work better in everyday tasks.
What this means in practice
- •For robotics engineers: Evaluate and improve spatial perception and task execution in robotic systems combining navigation and manipulation.
- •For automation developers: Assess integrated language understanding and physical task-solving abilities for developing advanced service robots.
Authors
Huapeng Li, Fuxiang Feng, Jinqiu Fan, Shuo Yang, Fengjiao Chen, Xuezhi Cao, Ran Song, Wei Zhang
Abstract
Although the advancement of vision-language models (VLMs) has endowed robots with enhanced environmental understanding and task reasoning, a comprehensive evaluation methodology is important to advance the integration of VLMs in robotic navigation and manipulation. However, current benchmarks lack a comprehensive method to evaluate diverse robotic tasks, and evaluation metrics remain relatively constrained, making it difficult to assess the embodied capabilities of VLMs in a thorough and fine-grained manner. To address this issue, we propose RMMBench, an evaluation benchmark that requires robots to understand language instructions and perform long-horizon tasks in continuous spaces. RMMBench seamlessly integrates high- and low-level embodied tasks into a unified framework, constructing a "navigation-manipulation" task suite comprising 70 canonical task scenarios that range from localized manipulation to long-horizon composite navigation. The results reveal that leading VLMs still face major challenges in spatial localization when performing mobile manipulation tasks, and also highlight the necessity of enhancing the spatial perception capability of robots during long-horizon interactions. RMMBench can be accessed at https://mxxq-stack.github.io/rmmbench-project/