Always-on robot manages dual arms in live changing environments
Watch, Recall, Act: Always-On Robots in Concurrent Embodied Streams
Robotics
Summary
Robots often get a fixed set of instructions and work step by step, but real life doesn’t work that way. The authors created a robot system that can watch a live video feed, remember its past actions, and control both arms at the same time without waiting. They built a new dataset to teach the robot how to do these things and showed it performs better than existing methods. This helps robots handle ongoing tasks that change over time without stopping.
What this means in practice
- •For industrial robot operators: Enable robotic arms to continuously handle complex tasks with changing instructions and environments without pausing or restarting processes.
- •For warehouse automation teams: Improve automation systems that must react in real time to new instructions and dynamic scene changes while coordinating multiple robotic arms.
Authors
Ding Yi, Peiwen Sun, Chenchu Rong, Jianan Wang, Xili Dai, Xiangyu Yue, Xi Lin
Abstract
An always-on robot faces an endless stream that never resets: instructions arrive and lapse, the scene changes, and its own past actions reshape what it must reason about. Today's action models are built for the opposite: a fixed instruction, no mid-task intervention, single-step reasoning. In an open-ended world a robot must watch a live stream for far-future cues, recall its own far-past actions, and act on them under dual-arm concurrency. We present ARMS (Always-on Robot in Multi-modal Streams), a deliberately simple streaming policy: a single pretrained $π$0.5 backbone augmented by three lightweight modules that turn live perception, embodied states, and the robot's own past actions into context the backbone reads before it acts. The modules update this context asynchronously, so watching and recalling never block acting and the two arms act at once. Rather than inventing new mechanisms, ARMS integrates these learned context providers with an agent-causal self-history that logs which arm did what, and when. To supervise them without extra annotation, we build ARMS Dataset, whose staged construction script itself labels every module from real dual-arm teleoperation. Trained on it, ARMS reaches 45% on the combined task against 28% for the strongest of our four main baselines, and ablations confirm the memory module, the embodied-state head, and asynchronous concurrency are each necessary.