Robots share compressed perception to communicate safely in teams
Latent Telepathy: Multi-Robot Communication with Self-Supervised Perceptual Latents
RoboticsMachine LearningMultiagent Systems
Summary
Robots working together need to share information about what they see to make good decisions. Instead of sending simple data like where they are, this paper has robots share a special small message made from their own understanding of what they observe. These messages are easy to understand across all robots and help them avoid dangers they can’t directly see. The authors show this method works better than sending raw images or expected motion paths, even on real robot cameras.
What this means in practice
- •For autonomous vehicle developers: Send compact perceptual messages between vehicles to improve hazard avoidance under limited visibility.
- •For drone fleet operators: Use frozen shared encoders to enable drones to share meaningful sensory information with minimal computational overhead and latency.
Authors
Howard Wang, Han Zheng, Cathy Wu
Abstract
In a decentralized multi-robot team under partial observability, the fact that decides a robot's next action is often visible only to a teammate. Existing decentralized methods communicate kinematic information, such as position or planned trajectory, which cannot convey what the teammate perceives. Learned communication in multi-agent reinforcement learning (MARL) can carry perceptual content, but the resulting messages are task-coupled and opaque. We propose Latent Telepathy. Each robot broadcasts the perceptual latent vector it already computes for its own use, the output of an encoder trained with a self-supervised joint-embedding predictive objective, frozen, and shared across the team. A teammate learns to act on it from task reward alone. Because the encoder already runs for perception, the message costs no additional computation and a single compact vector of bandwidth. Because the encoder is frozen before any policy is trained, the message means the same thing to every robot, and the receiving robot is never told what it means. We evaluate Latent Telepathy with a content-controlled protocol in which bandwidth, latency, topology and receiver are held fixed and only the message content varies. Broadcasting the latent lets a navigator avoid an occluded hazard in 99.7% of episodes, matching a noiseless hand-designed message. Position and trajectory messages remain at chance, and the raw camera image, 186 times wider, is less reliable than the compressed latent. The result holds from a discrete gridworld to rendered pixels under continuous velocity control, and the encoder decodes the hazard from a physical robot's camera in 102 of 102 live decisions. We also identify a requirement for porting MARL communication results to continuous control, that the decision a message informs must remain reachable by exploration, and show how to restore it.