Exact genomic tool selection improves policy training efficiency
Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection
Artificial Intelligence
Summary
Training AI policies to select the best tools often relies on guessing which tool combinations work well, which can be inefficient when all possible combinations can be checked. The authors found that in genomic reasoning tasks, where the possible tool sets are small enough to list completely, the usual method leads to poor learning because it keeps repeating the same guesses. They propose a new method that scores every tool combination exactly, speeding up training and improving performance. This method reduces the number of tools used per question and achieves better results across multiple genomic tests.
What this means in practice
- •For bioinformatics engineers: Optimize genomic reasoning pipelines by exactly evaluating all computational tool combinations, reducing resource use and improving prediction accuracy.
- •For machine learning engineers: Train reinforcement learning policies more efficiently when action combinations are enumerable by eliminating sampling noise and redundant evaluations.
Authors
Haoyue Liu, Xiaoyu Ma, Ye Chen, Zhichao Wang, Xiaoying Tang
Abstract
Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and (2) precomputes the reward of each question--subset pair into an exhaustive table, removing frozen-reasoner calls from the training loop entirely. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average and up to 14.20, while a standard on-demand GRPO schedule would require 2.4 times as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from 2.36 to 1.40.