Automatic harness tuning improves robot control from visual interfaces
Scale and Selection: What Makes Automatic Harness Evolution Work for Visual-Interface Robot Agents
RoboticsArtificial Intelligence
Summary
Robots that use computer programs to control their hands through a web interface rely on a setup called a harness, which includes instructions and tools. This setup was usually created by hand, but the authors showed it can be improved automatically by another program. The key discoveries are that using many examples to learn and carefully selecting changes leads to better and more reliable robot control. Without these steps, improvements can be accidental or lead to worse performance.
What this means in practice
- •For industrial robot programmers: Improve robot control policies that operate through visual web interfaces by automatically tuning the harness components for more reliable actions.
- •For software engineers for automation tools: Develop automated systems that refine control setups for robots interacting with software via screenshot-based visual inputs and virtual manipulators.
Authors
Zhijie Wei, Ferris Tan, Jinghui Wang
Abstract
When an off-the-shelf coding agent is used directly as a robot policy, observing a browser-based 3D interface through screenshots and acting by posing a virtual target gripper through a few tools, the agent's harness, its prompts, tools, and control rules, largely determines success, and until now it has been written by hand. We show that this harness can be improved automatically by another coding agent, the optimizer agent, and report two findings about what makes it work. First, the number of rollouts the optimizer agent sees per round governs whether the evolved harness is trustworthy, generalizes, and improves steadily. A single rollout is a noisy binary outcome, so with few rollouts per round a revision can be promoted on luck; enlarging the batch raises the signal-to-noise ratio of every promotion decision. Holding rounds fixed and growing the training set from 5 to 100 rollouts, held-out success rises from 47% to 67%, while small training sets overfit, reaching 70% on training tasks but only 54% held-out. Second, the optimizer agent must not be given free rein. With every revision it proposes accepted unconditionally, performance drifts downward within ten rounds as ill-judged edits accumulate; adding the most basic safeguard, Champion-Challenger selection that promotes a revision only if it strictly beats the incumbent on the same fixed evaluation set, turns the same loop into one that raises held-out success from 51% to 67% over 30 rounds. Automatic harness evolution for visual-interface robot agents is thus feasible, but its gains hinge on the rollout scale behind each decision and on how the optimizer agent's revisions are selected.