Negative-only learning improves reward verification in reinforcement systems
Negative-Only Policy Optimization for One-Sided Verifiable Rewards
Machine LearningComputation and Language
Summary
Many reinforcement learning systems need a way to check if their actions are correct, but sometimes only failure cases can be reliably identified, not successes. The authors identify this situation as one-sided verifiability, where only negative feedback can be trusted. They introduce a new method called Negative-Only Policy Optimization (NOPO) that trains systems by focusing on avoiding mistakes rather than confirming successes. Their approach improves performance on tests for coding, logic, and math tasks when perfect verification is not available.
What this means in practice
- •For software development teams: Improve automated code correctness checks when only failure cases can be confidently detected, enhancing testing workflows with partial verification feedback.
- •For machine learning operations teams: Refine training of AI models that verify their own outputs under uncertain or incomplete reward signals, enabling better performance without full oracles.
Authors
Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, Bo An
Abstract
Reinforcement learning with verifiable rewards (RLVR) typically relies on a verifier that can reliably judge whether sampled solutions are correct. However, in many realistic settings, complete verification is generally unavailable, while cheap checks can only falsify failures. We formalize this regime as one-sided verifiability (OSV): verified negatives are reliable counterexamples, but verified positives are merely non-falsified candidates and may contain many false positives. This asymmetry makes standard RLVR unreliable, because reinforcing verified positives can directly reward shortcuts that pass partial checks. To address this issue, we propose Negative-Only Policy Optimization (NOPO), which treats verified positives as unlabeled and updates the policy only by suppressing trusted verified negatives. Specifically, NOPO combines a reference-relative negative suppression loss with a group-level adapter that scales each update by the proportion of non-falsified responses. Across unit-test, constraint-check, and LLM self-verifiers, NOPO improves pass@k over baselines on code, logic, and math benchmarks when oracle rewards are unavailable. These gains persist under inference-time scaling and cross-benchmark transfer.