Human control of robot swarms through natural gestures and automatic code
Auto-HSI: Personalized human control of a robot swarm on demand by using LLMs for online automatic code generation
RoboticsArtificial IntelligenceHuman-Computer InteractionMultiagent Systems
Summary
Controlling groups of robots can be tricky, especially for people who aren't experts. This paper introduces Auto-HSI, a system that lets anyone use simple gestures and natural language to tell robots what to do together. The system then creates the computer code needed for the robots to respond exactly the way the user wants. The authors tested this on up to 50 simulated robots doing tasks like moving through mazes or scoring goals, and also showed it working with real robots.
What this means in practice
- •For robotics operators: Create custom control interfaces for robot swarms using natural gestures and language without programming knowledge.
- •For human-computer interaction designers: Develop interactive systems that generate executable control code from user demonstrations and descriptions for personalized device operation.
Authors
Alessandro Nazzari, Nathan Cerisara, Dorian Tonnis, Raina Zakir, Lorenzo Labarile, Weixu Zhu, Marco Dorigo, Mary Katherine Heinrich
Abstract
This paper presents Auto-HSI, a method for generating personalized human-swarm interaction (HSI) interfaces on demand. The objective is to enable untrained operators to use natural language descriptions and gesture demonstrations to explain how they want the robots to collectively behave in response to their gestures. Based on these inputs, the code should automatically be generated for personalized state machines that will control the robots as desired, in response to the desired gesture inputs. In the developed Auto-HSI prototype, the generated code produces a personalized interface for centralized control using one- and two-handed gestures, enabling a user to teleoperate the robots' motion, formation shape, and shape deformation. We test the gesture tracking and code generation components of Auto-HSI against performance benchmarks. We then test the full Auto-HSI prototype in ``live'' operation experiments, in which real human operators centrally control 50 simulated robots in a physics-based simulator, under nominal and noisy conditions. In these experiments, robots are teleoperated to: score a goal, traverse a maze that requires shape deformation, and score two simultaneous goals by splitting into two groups. We also demonstrate a real human operator making live updates to their personalized Auto-HSI interface during operation (in simulation). Finally, we demonstrate live operation of real robots.