RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning
2026-07-20 • Robotics
Robotics
AI summaryⓘ
The authors created RoboHarness, a system that helps robots handle complex tasks by combining different specialized robot control policies. Instead of relying on one type of skill, RoboHarness figures out when and how to switch between various policies based on how well they work in different situations. It uses memory of past executions and real-time information to guide these switches smoothly, without needing to retrain policies together. Tests show that this approach improves robot planning for long and varied tasks, even in new or unexpected conditions.
robot policieslong-horizon tasksmulti-modal execution memorytask-and-motion planning (TAMP)reinforcement learning (RL)policy handoffcapability boundarieszero-shot planningpolicy orchestrationout-of-distribution robustness
Authors
Jinbang Huang, Yuanzhao Hu, Zhiyuan Li, Ran Qi, Yixin Xiao, Zhanguang Zhang, Mark Coates, Tongtong Cao, Yingxue Zhang
Abstract
Long-horizon robotic tasks require diverse capabilities that no single policy can reliably provide. Heterogeneous policies offer complementary strengths, but orchestrating them requires reasoning over uncertain capability boundaries and cross-policy distribution mismatch, which are largely overlooked by existing planning methods built on homogeneous, predefined skills with fixed applicability. We propose RoboHarness, a unified framework that encapsulates independently developed robot control systems as reusable agentic skills. Although instantiated in this work with VLAs, RL policies, and task-and-motion planning (TAMP) systems, RoboHarness is designed as a general framework compatible with a broader range of robot policies, such as navigation policies, model predictive controllers, and world-action models. RoboHarness uses multi-modal execution memory and online evidence to characterize policy capability boundaries for capability-aware decomposition and routing. To stabilize policy handoffs, its Memory Bridge retrieves execution trajectories associated with the next policy, estimates its in-distribution state region, and guides the robot toward that region without joint policy retraining. Extensive experiments on three public benchmarks, 500 customized tasks, and 135 real-robot experiments demonstrate effective capability-aware routing and stable policy orchestration, yielding substantial improvements in zero-shot long-horizon planning and out-of-distribution robustness.