Maniskillformer enables robots to manipulate diverse objects without demonstrations
ManiSkillFormer: Demonstration-Free Compositional Manipulation via Task-Conditioned Geometric Contracts
Robotics
Summary
Robots often need many examples to learn how to handle different objects, which takes a lot of time. This paper presents ManiSkillFormer, a system that helps robots perform various tasks without needing those example demonstrations. It breaks tasks into clear geometric parts and uses large language models to plan how to interact with objects of different shapes. ManiSkillFormer lets robots pick up, use, and combine skills on many objects without retraining. The authors tested it on a real robot with strong success across a range of tasks.
What this means in practice
- •For robotics engineers: Build versatile robots that perform multiple object tasks without retraining on each new item.
- •For industrial automation teams: Deploy robots capable of compositional manipulation like unscrewing and pouring across different products without extra demonstrations.
- •For warehouse automation developers: Create picking systems that handle a wide variety of packages and items without per-object customization.$Commercial implications: This paper enables development of adaptable warehouse robots selling automation solutions to logistics providers.
Authors
Peiqi Yu, Mosam Dabhi, Shangtao Li, Bowei Li, Laszlo Jeni, Changliu Liu
Abstract
We present ManiSkillFormer, a neuro-symbolic framework for demonstration-free and compositional robotic manipulation. Instead of learning end-to-end visuomotor policies, ManiSkillFormer introduces task-conditioned geometric contracts that explicitly structure the interface between perception and action. Each manipulation skill declares the semantic geometric primitives required for execution, such as object keypoints and surface normals. Building on human-defined skill structures, LLM agents generate these contracts and corresponding motion templates for different objects and task contexts. These contracts guide the perception module to ground task-relevant 3D primitives from observations, which are then used to instantiate reusable motion templates stored in a skill library. We evaluate ManiSkillFormer on Galaxea R1-Lite dual-arm robot across three settings: zero-shot pick-and-place over 8 object categories with 30 different instances, functional manipulation tasks including unscrewing, pouring, pressing, and folding, and 3 long-horizon tasks. ManiSkillFormer achieves higher average success rates than the evaluated baselines and two ablated pipelines: 88.24% for demonstration-free pick-and-place, 75.00% average success on functional manipulation and 50--80% completion rates across the long-horizon tasks. These results show that our design enables composable and reusable manipulation across objects and tasks without per-object policy fine-tuning or additional robot demonstrations.