Papers for

smart factory designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multi-robot teams learn to talk and work together in kitchens

RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations

Abstract: Multi-robot collaboration could enable more efficient and scalable solutions to complex robotic tasks, but collaboration under partial observability remains challenging. Natural-language communication offers a promising approach to coordinating robots under partial observability. However, in decentralized manipulation, jointly learning explicit inter-robot communication and skill-level action selection from multimodal demonstrations remains underexplored for small vision-language models (VLMs) intended for on-device deployment. To address this gap, we introduce RoboTalk, a synthetic data-generation pipeline and dataset of 7,950 multimodal trajectories spanning 53 mobile-manipulation kitchen tasks for training small VLMs to communicate and coordinate. The dataset includes a leader-follower planning protocol, tool calls (perception, manipulation, navigation, and communication), rationale traces, and diversified natural-language communication. Fine-tuning open-source models on our dataset can reach 77% success on novel held-out tasks, a significant improvement over the untuned open source models, which had a success rate of around ~2%.

Mon 21 SeptRoboticsArtificial Intelligence
The gist
Multi-robot cooperation is hard when robots can’t see everything around them. The paper presents RoboTalk, a way to teach small robots to communicate using simple language and coordinate their actions by learning from example videos and instructions. RoboTalk creates a big set of example tasks where multiple robots work together in kitchen scenarios, sharing plans and talking to each other. Training robots this way helps them succeed much more often on new tasks. The authors show that fine-tuning existing language-vision models with their data improves robot teamwork from almost never to more than three-quarters success.
Open → 2609.23997v1