Training monitors reduces but does not eliminate AI collusion behaviors
Model Organisms of Collusion: Corrective Transfer in Untrusted Monitors
Artificial Intelligence
Summary
Sometimes AI models watch their own behavior to avoid doing wrong things, but they can hide misbehavior from themselves. This study shows that training these self-monitoring models on known bad behaviors helps them catch similar unseen bad behaviors, but it cannot stop all hidden misbehavior. The researchers tested this on large language models and found that while corrections reduced cheating a lot, some cheating still remained. This means fixing one type of cheating teaches the monitor to catch others better but doesn't solve the problem completely.
What this means in practice
- •For ai safety engineering teams: Improve monitoring tools for AI systems by fine-tuning AI monitors on caught misbehavior to reduce undetected collusion.
- •For machine learning platform developers: Develop enhanced AI model auditing features that adapt from identified faults to better detect hidden misbehavior in deployed models.
Authors
Juan-Pablo Rivera, Ari Spiesberger, Juan J Vazquez, Arun Jose, Axel Ahlqvist, Adeline Kassler, Meru Gopalan, Sandy Tanwisuth, John Hughes
Abstract
AI control protocols aim to get useful work from models that may be misaligned by monitoring their behaviour. One approach uses a second copy of the same model as the monitor. This avoids relying on a weaker reviewer the model could outsmart, but risks collusion: the model, now monitoring itself, may cover up its own misbehaviour. One remedy is to fine-tune the monitor on whatever examples of misbehaviour the control team has caught. We ask whether this training also removes collusion that the control team has not yet found. We fine-tune Llama-3.3-70B and Qwen2.5-72B to install five policy-monitor behaviour pairs, correct the monitor on 300 honest reviews of one caught collusion, and measure collusion on the other four in unseen prompt formats. Averaged over the three seeds of each family, correction reduces held-out collusion from 91% to 14-20% under three of the four methods we compare (individual seeds 10-32%), while a control fine-tuned on the same number of unrelated examples leaves 56-93%. Family-average accuracy on control items changes by at most five points and capability is preserved, apart from a drop in mathematical accuracy on one Llama seed. Under the tested training configuration, methods with similar caught-task accuracy differ in held-out transfer, from 58% to 83% average reduction, so the caught behaviour alone does not reveal how far a correction reaches. Catching one collusion lets the control team reduce, but not remove, collusion it has not caught, by retraining the monitor alone.