Audit First VAPO improves update safety control in AI systems
Audit-First VAPO: Risk-Certified Selective Updates under Imperfect Verification
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Sometimes AI systems make updates that unintentionally harm their performance, even if these updates are small. The authors present Audit-First VAPO, a method that checks the safety of update directions before allowing them and controls the size of the update afterward. This approach uses a limited secondary check to minimize risk and provides statistical guarantees about harmful update rates and how often checks are needed. Their tests on reasoning benchmarks showed that this method reduces harmful updates compared to random or simpler selection approaches while keeping performance high.
What this means in practice
- •For machine learning engineers: Improve the safety of iterative model updates by selectively admitting updates based on audit policies that limit harmful risk and verification costs.
- •For automated reasoning teams: Apply statistical risk certification to control update reliability in AI models trained for benchmark reasoning tasks.
Authors
Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Rui Chen, Daren Zha, Jun Xiao
Abstract
Imperfect verifiers can assign a harmful update direction even when clipping and regularization bound its magnitude. We introduce Audit-First VAPO, which separates discrete directional admission from continuous magnitude control. An observation-only accept-appeal-abstain policy uses a finite secondary-verification budget; its action trace is frozen before clean labels are joined. Simultaneous finite-sample bounds then certify selected harmful risk, coverage, and verifier-call rate over a predeclared policy family. Conditional Hoeffding-Azuma bounds account for the dependence induced by shared budgets, and rollout or verifier changes initiate a new certification stage. After admission, a bounded trust-clip-KL actuator controls magnitude. We evaluate two models on two reasoning benchmarks against static RLVR, matched-random selection, confidence thresholding, noise correction, and verifier augmentation. On Qwen3.5-0.8B and GSM8K at target risk $ρ=0.08$, RC-VAPO achieves 74.1% accuracy, selected harmful risk 0.0697, coverage 0.4125, and relative verifier cost $1.16\times$. At matched coverage and update magnitude, its selected-risk difference from matched random is -0.0260 with paired 95% interval $[-0.0364,-0.0157]$. Across asymmetric, confidence-dependent, and correlated-verifier noise, the certificate is satisfied on 57 of 60 independent runs. These comparisons isolate informative directional selection from proposal suppression, update shrinkage, and additional verifier computation.