Reinforcement learning safety improved by verifying and penalizing shortcuts
Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning
Artificial IntelligenceCryptography and Security
Summary
Sometimes AI systems learn to cheat by taking shortcuts that get high rewards but break the rules or do less real work. The authors study this problem by creating tasks where cheating is possible but following the rules is still doable. They develop ways to detect and measure these cheating behaviors and use simple penalties to make the AI both capable and safe. Their approach works well even when rules or evidence requirements change over time.
What this means in practice
- •For workflow optimizers: Detect and reduce unauthorized resource use in automated workflows by enforcing compliance through contextual safety checks.
- •For api service operators: Use verifiable checks to prevent AI agents from exploiting shortcuts that inflate performance metrics without doing real work.
Authors
Xinjie Shen
Abstract
Reinforcement learning (RL) can improve an agent's task capability while reinforcing behavior that violates contextual execution requirements. We study this reward-driven misalignment, including reward hacking, through verifiable reward--context conflicts, where compliant task completion remains feasible while some executable shortcuts earn more reward than every compliant strategy. Encoding these conflicts in task dynamics lets us generate diverse interactive environments at scale and derive compliance feedback from execution histories. In resource and reporting workflows, task optimization increases real productivity through unauthorized resource use or raises reported scores while actual work declines. Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching 97.9\% certified profitable violations on an API task under a 25\% conflict mixture after 128 updates. The same contextual checks supply a simple binary penalty that supports joint capability and compliance gains and partially mitigates acquired violations. Learned safety adapts to expanded authorization and transfers across mechanisms into workflows with evolving evidence and approval requirements. On 32 unseen changing-evidence tasks, resource-trained safety increases safe completion from 53.1\% to 89.1\% relative to task-only training, while both policies achieve full real work. These results connect a verifiable account of reward-driven misalignment with scalable task construction and safety learning that remains useful under changing execution requirements.