HarnessBandit improves multi-harness training for language model agents
HarnessBandit: Joint Learnability-Transferability Scheduling for Multi-Harness Agentic Reinforcement Learning
Machine LearningArtificial IntelligenceComputation and Language
Summary
Language model agents work differently depending on how they are connected, which makes it hard to train them well for all situations. The authors introduce HarnessBandit, a method that picks which connection (called a harness) to train on at each step based on how much the model is learning and how well the update helps other connections. Their approach balances learning new things and sharing improvements, leading to better overall performance on different tasks. They tested it on robotic simulation tasks and showed it works better than simply mixing all connections together during training.
What this means in practice
- •For robotic system developers: Improve training efficiency and robustness of robot control policies that interact through varied software interfaces.
- •For machine learning engineers: Select training tasks dynamically to enhance shared policy learning across diverse environments or input-output formats.
Authors
Hongliang Wei, Xiaobing Tu, Yinggui Wang, Zhengxi Liu, Rongkun Xue, Jinkui Ren, Xiantao Zhang, Debin Zhao, Xiaopeng Fan
Abstract
Language-model agents are increasingly deployed through diverse harnesses that differ in system prompts, tool schemas, control loops, and trajectory formats. The same model can perform unevenly across these interfaces, making robustness to harness variation an important objective. A natural approach is to train a shared policy through multiple harnesses, but doing so introduces a scheduling problem: each training step should favor a harness that currently provides a useful learning signal while also producing an update that benefits the other harnesses. We develop HarnessBandit, an online scheduler that selects one harness per optimizer step. After a group-relative policy optimization (GRPO) update, it observes learnability -- the mean absolute advantage on the batch -- and transferability -- the cosine between a low-dimensional gradient sketch of the current harness and exponential moving averages of the remaining harnesses. The two signals are fused after pooled sliding-window min-max normalization and sampled with a visit-dependent bonus and an explicit exploration floor. We train Qwen3.5-2B across six harnesses on ClawGym and evaluate on PinchBench (held-out tasks, in-distribution OpenClaw) and ClawEval (held-out tasks and harness). HarnessBandit improves over mixed-batch multi-harness training on both benchmarks, while training diagnostics indicate that learnability and transferability provide distinct, evolving signals.