FineART dataset advances robot bimanual task learning and control
FineART: Fine-grained Annotated Robotic Trajectory Dataset and Vision-Language-Action Model for Bimanual Manipulation
RoboticsArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Robots often need to use both hands to carry out complicated sequences of tasks, but existing data about these actions is limited or incomplete. The authors created FineART, a large dataset with detailed labels for many bimanual robot tasks, and also built a new AI model called FineART-VLA that learns to predict the next small step in these tasks. This approach significantly improves robots’ ability to understand spatial instructions and perform long sequences of actions, even on new robots and tasks they have not encountered before. The dataset and tools are released to help others improve robot manipulation.
What this means in practice
- •For industrial robot programmers: Train robots to perform complex two-handed tasks more reliably with detailed subtask guidance.
- •For robotics software developers: Develop robot control software that adapts quickly to new hardware and unfamiliar tasks with less data.
Authors
Jade Choghari, Pepijn Kooijmans, Mansi Agarwal, Yusuf Umut Ciftci, Aseem Doriwala, Catherine Weaver, Mouli Sivapurapu, Kai Yang, Jackson Lee, Thomas Wolf, Pragna Mannam
Abstract
Robots operating in real-world environments must execute complex, multi-step bimanual tasks over long horizons rather than single, isolated actions. Current manipulation datasets struggle to support this capability: although single-arm datasets reach hundreds of thousands of trajectories, they typically provide only one high-level instruction per episode while the rare bimanual effort that does label subtasks annotates only a fraction of its hours. We present FineART, a densely annotated bimanual manipulation dataset of 40,543 episodes, 1,718 hours, and 533,913 subtasks across 151 tasks. We also introduce FineART-VLA, a vision-language-action policy that predicts its own next subtask, and show that mid-training it this way yields substantial gains. Specifically, success on a spatial disambiguation task increases from 32.0% to 100.0%, and step-by-step human subtask guidance lifts success on an unseen long-horizon task from 16.0% to 76.0%. Furthermore, after minimal fine-tuning on a new robot, the policy requires one-tenth the data of baselines without mid-training and generalizes zero-shot to completely unseen tasks on the new hardware. We open-source the full dataset, model weights, and training code.