Large language models tested controlling real robots and sensors
Words to World: Benchmarking LLMs in Interactive Physical Environments
Artificial Intelligence
Summary
It’s becoming common for AI language tools to control machines, but we don’t know how well they do in real physical settings. The authors created a test where language models operate real robots and sensors, adjusting their actions based on actual feedback. Their tests reveal challenges like misinterpreting what happens in the world and trouble fixing mistakes. This work helps show how well AI can turn words into physical actions with real-world effects.
What this means in practice
- •For robot developers: Evaluate language models’ ability to send instructions and adapt to real-time sensor feedback in controlling physical robots.
- •For industrial automation teams: Test AI-driven control systems for robust decision-making and error recovery in complex physical environments with real hardware.
- •For smart home device makers: Improve AI assistants to manage multiple devices using natural language while adapting to unpredictable real-world conditions.$Commercial implications: Enables selling AI-powered home automation systems that reliably handle real-world device interactions based on language.
Authors
Xinnan Dai, Kaiqi Yang, Yaxin Li, Cheng Luo, Wanli Li, Mingqing Xiao, Xinyang Jiang, Daoan Zhang, Kaitao Song, Fengze Liu, Zefan Cai, Caihua Shan
Abstract
Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that affect the physical world. However, current benchmarks largely evaluate LLMs through static text responses, offline tool calls, or simulated tasks, leaving open the question of how well they can act in real physical environments. We present a benchmark for measuring LLMs in closed-loop physical interaction, where models control a harness system connected to real-world components including robotic arms, cameras, and sensors. In each task, the model must issue instructions, receive physical feedback from the environment, interpret the resulting state, and adapt its next action accordingly. This setting evaluates capabilities beyond language understanding, including grounded perception, action planning, tool selection, spatial reasoning, feedback-driven correction, and robustness to execution uncertainty. Through experiments with representative LLMs, we characterize their ability to translate linguistic reasoning into physical actions and identify common failure modes such as hallucinated state changes, poor recovery from unexpected outcomes, and unstable long-horizon plans. Our benchmark aims to provide a systematic foundation for evaluating LLMs as interactive agents that connect words, tools, and the physical world.