CompassPlay rewards task proposers for guiding solver learning effectively
CompassPlay: Rewarding the Proposer for Where It Moves the Solver
Machine Learning
Summary
Training AI to solve problems often involves generating tasks for it to practice on. However, not all tasks of the same difficulty help the AI learn equally well. The paper introduces CompassPlay, a technique that rewards the task creator based on how well the tasks improve the solver's learning direction. This method uses gradient alignment to pick tasks that better match target skills without extra training effort. Tests showed CompassPlay improved accuracy in coding and math tasks and reduced resource use in theorem proving compared to standard methods.
What this means in practice
- •For ai model trainers: Increase training efficiency by guiding task creation to tasks that boost solver learning without extra solver retraining.
- •For software engineers: Improve automated coding task generation using small reference sets to better train code generation models in domain and out-of-domain scenarios.
Authors
Sophia Xiao Pu, Ximeng Sun, Jiang Liu, Jialian Wu, Emad Barsoum, Zicheng Liu, William Yang Wang
Abstract
In self-play, a proposer generates verifiable tasks to train a solver. Proposer rewards often depend on the solver's success rate, but equally difficult tasks can differ in their training value. We introduce CompassPlay, a self-play method that rewards the proposer through gradient alignment. The reward favors tasks whose solver loss gradients align with those of reference tasks representing the target capabilities. It draws on a first-order approximation to learning progress and scores each eligible task without additional solver training. Our experiments show gains in performance and training efficiency. In coding self-play with Qwen2.5-Coder-7B, a small external reference set guides task generation. CompassPlay improves average accuracy over AZR's difficulty reward by 1.5 percentage points on in-domain coding and 2.7 on out-of-domain mathematics. In Lean4 theorem proving, CompassPlay matches the difficulty baseline's 150-iteration cumulative coverage with 40\% fewer GPU-hours.