Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulation
2026-08-24 • Robotics
RoboticsArtificial Intelligence
AI summaryⓘ
The authors present the Triplet-to-Track System (TTS), a method to help robots perform long, complex tasks more reliably by learning from human videos instead of lots of robot data. Their system breaks down goals into simple parts tied to real objects, keeps track of progress during the task, and can adjust actions on the fly. This approach improves robot reliability and understanding during tasks, achieving a nearly 75% success rate in real-world tests and adapting to new objects and task setups.
imitation learningrobotic manipulationlong-horizon taskshierarchical planningclosed-loop controlobject groundingonline replanningend-to-end modelstask monitoring
Authors
Jianxiang Liu, Gaojing Zhang, Chuan Wen, Qipeng Liu, Yuxuan Zhao, Ning Guo, Wenzhao Lian
Abstract
Ensuring reliability in uncertain environments remains difficult for long-horizon robotic manipulation. End-to-end VLA models are data-heavy and opaque, making diagnosis and verification difficult. Hierarchical pipelines are more interpretable, but their plans are often weakly grounded in observations, weakly aligned with low-level actions, and computed without online feedback, leading to open-loop behavior and hallucinations. To address these issues, we introduce the Triplet-to-Track System (TTS), a closed-loop long-horizon imitation learning system that uses human videos to reduce reliance on robot-collected data. TTS represents high-level subgoals as instance-grounded triplets, translates them into continuous track priors for execution, and monitors task progress from observations for online replanning. Across diverse real-world long-horizon tasks, TTS achieves a 74.8\% average success rate and supports object-level and compositional generalization.