Large language models differ in robot safety compliance and behavior
How Long Until Your Robot Ignores You? A Safety Benchmark for LLM Orchestrators in Human-Humanoid Collaboration
Robotics
Summary
Robots guided by large language models (LLMs) need to act safely around humans, but there was no way to test how well these AI systems keep people safe. The authors created the first safety test for AI that controls humanoid robots, checking if the robot follows safety rules, refuses unsafe actions, or disobeys them altogether. They tested several popular LLM systems and found big differences: some rarely broke safety rules, while others made more mistakes. The way the AI remembers conversation context also affected safety behaviors in surprising ways.
What this means in practice
- •For robotic system designers: Use the benchmark to evaluate and improve safety-aware behavior of AI-based robot controllers in human-robot collaboration settings.
- •For industrial automation engineers: Assess different language models for deploying humanoid robots safely in industrial environments following ISO protective standards.
Authors
Aulon Bajrami, Mohamed Elshamouty, Werner Kraus
Abstract
Large Language Models (LLMs) are increasingly employed to orchestrate robot behavior through natural-language interfaces, yet no benchmark exists to evaluate their reliability as safety-aware decision makers in human-humanoid collaboration. Unlike deterministic safety systems that enforce binary allow/deny decisions, LLM-based orchestrators exhibit a compliance spectrum ranging from overcompliance (refusing safe actions) to full safety violations. This paper introduces the first safety benchmarking environment for LLM orchestrators in human-humanoid collaboration, built on a Model Context Protocol (MCP)-based architecture with safety invariants grounded in ISO 10218-2:2025 protective measures. The benchmark defines five testable safety invariants, a four-level compliance taxonomy (correct compliance, overcompliance, undercompliance, full violation), and a three-layer evaluation pipeline (text prompting, simulated sensor-actuator loops, and physical validation on a Unitree G1 EDU humanoid). We report Layer-1 results: three cloud backends (Claude Haiku 4.5, GPT-4o-mini, Gemini 2.5 Flash) and a local open-weights baseline (qwen3:8b) across 40 100-turn sessions under full-context and sliding-window budget conditions, while the simulation and physical layers remain ongoing. We find that (1) model family determines the safety floor, as Claude and Gemini remain at or near zero violations while GPT-4o-mini commits up to 13 per session, (2) context management dissociates two failure axes, reducing mean behavioral issues by 42-57% for every cloud backend while nearly doubling GPT-4o-mini's violations (3.8 to 7.2 per session), and (3) proportional compliance, clamping movement speed to the rule-specified maximum rather than refusing, emerges consistently only in Gemini; the preliminary simulation layer reproduces the model ranking and the GPT-4o-mini failure-mode inversion.