Dataset supports robots using past events to guide actions
Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Sometimes robots must remember what happened earlier to decide what to do next, because just looking around doesn’t give all the answers. The authors created a big collection of robot task examples where they include notes about what happened before, so robots can learn when history matters. They tested existing robot models and found these models still struggle with remembering past events to make better decisions. This new dataset and testing setup help improve robots that do complex tasks needing memory.
What this means in practice
- •For robotics development teams: Test and improve robot control policies that must use past event information across biolaboratory, household, and industrial tasks.
- •For automation engineers: Evaluate how well robots perform history-sensitive manipulation tasks to enhance reliability in complex real-world workflows.
Authors
Yi Wang, Yang Yang, Guangqi Xu, Sumin Lin, Ning Kang, Pengxiang Lu, Xiaotong Chen, Zeyu Xue, Ping Deng, Xing Liu, Chenguang Yang, Zhenyu Lu
Abstract
Robotic manipulation often requires inferring task-relevant states from past interactions when the current observation alone is insufficient to determine the appropriate action. Despite progress in benchmarking memory-augmented vision-language-action (VLA) models, application-oriented tasks requiring history-dependent semantic inference remain underrepresented. We introduce GiT (Grounded in Time), a dataset and benchmark for grounding manipulation decisions in past events across biolaboratory, household, and industrial scenarios. It includes real-robot and Universal Manipulation Interface (UMI) style demonstrations covering 18 bimanual tasks, together with simulation data and a ManiSkill-based evaluation suite covering nine tasks. Fine-grained subtask annotations and annotated counterfactual task pairs, in which similar current observations require different actions depending on prior events, support policy learning and targeted evaluation of history use. Evaluations of representative end-to-end VLA models in simulation and on selected real-world tasks reveal substantial room for improvement in history-dependent manipulation. The dataset and benchmark are available at the project page.