Robots benchmarked for real-life public environment tasks
Benchmarking Robots for Everyday Environments: From Lab Experiments to Real-World Operations
RoboticsHuman-Computer Interaction
Summary
Robots are often tested in labs, but this doesn't fully show how well they work in everyday places. The authors studied three different robots doing jobs like cleaning parks and helping in libraries in real outdoor and indoor spots over three years. They checked not just if the robots did their jobs, but also how safe and easy they were to use around people and if running them made economic sense. Experts from different fields worked together to decide how to judge the robots fairly during this long testing process. The results highlight what matters most for robots to succeed when they leave the lab and start working where people live and play.
What this means in practice
- •For robotics development teams: Design and test robots with clear metrics that reflect real-world tasks and environments to improve deployment success.
- •For smart city planners: Use validated benchmarks to evaluate and select robotic solutions for cleaning and assistance tasks in public areas.
Authors
Raphael Memmesheimer, Martina Overbeck, Dominik Beyer, Björn Kral, Sabine Bellmann, Sven Schneider, Jan Zimmermann, Anna-Maria Meer, Medina Klicic, Simone Roth, Carolin Straßmann, Alexander Arntz, Marlene Wessels, Johannes Kraus, Paul Schweidler, Tristan Schnell, Christoph Zimmermann, Benedikt Pulver, Wilhelm Stork, Martin Gersch, Sven Behnke, Arne Rönnau
Abstract
This study introduces an interdisciplinary framework for benchmarking robots deployed in public environments, addressing the gap between traditional laboratory metrics and real-world benchmarking requirements. We evaluate three distinct robots across diverse use cases - outdoor park cleaning, pedestrian underpass cleaning, and interactive library assistance - each representing unique challenges in public daily life. Over a three-year benchmarking process (2023-2025) comprising seven benchmarking events, a consensus workshop and six on-site evaluations (two per use case), we utilized realistic indoor and outdoor test environments to assess not only technical performance but also the broader implications of deploying robots in unstructured, human-centric settings. An expert panel, spanning robotics, human-robot interaction, safety, and economics, systematically developed and refined an evaluation concept to analyze the transition from laboratory prototypes to operational systems. Our findings highlight critical factors for successful deployment, including task fulfillment, interaction quality, safety, and economic feasibility. This work provides actionable insights for researchers and practitioners aiming to bridge the gap between robotic innovation and real-world applicability.