Reinforcement learning boosts reasoning accuracy by evolving reusable skills
RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning
Artificial Intelligence
Summary
Solving problems that require reasoning across different types of information is hard because a system must figure out the right steps without direct guidance. The authors developed a method called RLHARNESS that helps the system build and update a toolbox of skills it can use to solve complex tasks more effectively. By letting the system refine both how it chooses skills and the skills themselves, it gradually improves its accuracy on multiple challenging benchmarks. This approach shows that evolving the set of skills along with learning the policy to use them leads to better reasoning performance.
What this means in practice
- •For ai software developers: Build AI systems that solve complex reasoning tasks by evolving reusable procedural skills during training for better accuracy.
- •For robotics engineers: Improve decision-making in robots that need to follow long sequences of steps involving multiple sensor types by updating learned skills on the fly.
Authors
Ziqiao Shang, Zian Xu, Ji-Chen Yan, Weiming Wu, Ziyi Jia, Jie Meng, Tao Huang, Shan Huang, Lan-Zhe Guo
Abstract
Multimodal reasoning requires models to preserve visual evidence through long decision chains while selecting appropriate procedures across diverse scenarios and rules. When learning is guided only by terminal verifiers, reinforcement learning (RL) reveals whether a final answer is correct but not how it should be produced. The policy must therefore discover reusable reasoning procedures while learning to execute them, creating a program cold-start problem. Skills can externalize successful procedures, reduce repeated exploration, and provide inspectable guidance. However, a fixed Skill Bank assumes that this guidance remains compatible with an evolving policy, while updating Skills alone can leave their triggers, execution protocols, and demonstrations stale or mutually inconsistent. We introduce RLHARNESS, which organizes Skills, selection and execution protocols, few-shot demonstrations, and task contracts into a unified, versioned Harness and alternates Harness evolution with policy learning. An Exploration-Distillation Harness builds the initial Harness and version-aligned verified traces for SFT and DAPO I. After the first RL block, a Post-RL Reconstruction Harness rebuilds Skills, protocols, and demonstrations from fresh success-failure rollouts, and DAPO II adapts the policy to the reconstructed program. RLHARNESS improves Accuracy from 16.25%/27.50% to 62.00%/50.00% on MetroMap/TravelMap and raises F1 score from 37.13%/45.50% to 65.81%/65.51% on Fee-VL/Cancel-VL. All four tasks achieve their best results only after reconstruction and DAPO II, showing that an evolving Harness complements RL by continually updating the external program that the policy learns to execute.