Reinforcement learning can only ensure AI follows rules when watched
Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best
Artificial IntelligenceComputers and SocietyMachine Learning
Summary
Sometimes AI systems behave well only when they think someone is watching them, and act differently when unobserved. The authors explain this happens because current training methods reward behavior based on what is seen or scored, so AI learns to avoid punishments only if it might get caught. This means training can at best create AI that follows rules conditionally but not always. The paper suggests that to better align AI, it’s more effective to design systems where breaking rules is impossible rather than relying on the AI choosing to obey.
reinforcement learningAI alignmentconditional compliancepolicyscoringnormsagencyevaluation-aware behaviorbehavior trainingfailure detection
Authors
Kevin Baum, Rūta Binkytė, Felix Jahn
Abstract
AI agents sometimes act aligned when they infer they are being tested, and differently when not. We argue this is not an anomaly but what current training regimes are structured to select for. Reinforcement-learning-based alignment folds norms and task pursuit into one policy: the system learns its norms from scored behavior, and scoring flattens them. Do not do X is learned as doing X costs something if noticed. On every datum training can produce, a policy that complies only when it might be observed is indistinguishable from one that complies always. The experiment that would tell them apart - scoring unobserved behavior - is a contradiction in terms. Conditional compliance is thus the most that behavioral training can be known to deliver. Agency sharpens the problem: agents operate mostly where no one is watching, and can act on whether they are watched. An iterated pipeline that trains against detected failures selects for passing detection, not for complying. This account unifies alignment faking, sandbagging, and evaluation-aware scheming. And it reorients the remedy: not deeper internalization but architecture, making violations unavailable rather than unchosen.