Robots share task-focused views to improve teamwork in limited communication

Be My Eye: Learning to See Beyond a Robot's Own Perspective

RoboticsArtificial Intelligence

Summary

Robots working together often can't see everything they need because they each have their own limited viewpoint. The authors created a way for robots to share only the most important, task-related pieces of what they see so they can make better decisions together. Their system teaches robots to compress their observations into small messages that still contain key info about the task and progress. This helps pairs of robots work more effectively even when they can only send small amounts of data to each other. They tested this on real robots completing tricky tasks and saw better cooperation.

What this means in practice

  • For industrial robot teams: Coordinate dual-robot tasks more efficiently by sharing only task-relevant visual information under limited wireless bandwidth.
  • For autonomous drone operators: Enable pairs of drones to share compressed, task-focused views for improved collaboration during complex inspections or deliveries.

Authors

Baifu Huang, Shilin Fang, Jianan Wang

Abstract

Robots operating in the real world are inherently subject to partial observability, particularly in multi-robot collaboration, where individual observations may be insufficient for effective task execution. A key challenge is leveraging complementary observations from partner robots to support collaborative decision-making. We argue that information sharing should go beyond passively transmitting raw observations or generic visual features. Instead, robots should selectively encode task-relevant information according to task objectives and execution progress. To this end, we propose a task-oriented active context sharing framework that transforms robot observations into compact representations, preserving critical information under constrained communication budgets. The framework adopts a two-stage training paradigm. In the first stage, onboard observations and task instructions are jointly encoded into compact context representations. Task-oriented future prediction and soft task-progress correspondence learning guide this collaboration-agnostic compression, preserving information about task objectives and execution progress. In the second stage, the learned representations are integrated into a pretrained Vision-Language-Action (VLA) policy, enabling bidirectional context exchange. Each robot combines complementary partner information with local observations to guide action generation. Action supervision jointly refines the context representations and policy, aligning information sharing with collaborative decision-making. Controlled experiments demonstrate substantial performance gains under constrained communication budgets, while three real-world experiments validate the framework in complex dual-robot tasks. These results highlight a shift from passive visual transmission toward active, task-oriented context sharing that directly supports collaborative decision-making.