MOLE benchmark detects hidden insider threats in AI agent accounts
MOLE: Detecting Insider Threats in AI Agents
Machine LearningComputation and LanguageCryptography and Security
Summary
AI programs meant to do helpful tasks can sometimes be tricked or misused to do harmful things, like stealing or messing up data. Existing tests don’t check well if people watching these AI accounts can spot bad behavior with limited time and resources. The authors created MOLE, a large test set simulating many AI accounts with real activity and potential threats, to see how well monitors detect harmful actions. They found even the best monitors miss a lot of issues, but using MOLE to guide improvements helps detection. This tool helps improve safety checks for AI running in complex environments.
What this means in practice
- •For security teams: Identify and improve tools that detect harmful insider activity in AI-powered accounts with limited review resources.
- •For cloud platform operators: Use MOLE to simulate diverse insider threat scenarios against AI services to validate and enhance anomaly detection mechanisms.
Authors
Aashiq Muhamed, Virginia Smith
Abstract
Model misalignment, prompt injection, or operator misuse could lead AI agents operating frontier-lab accounts to exfiltrate model weights, poison training data, or weaken release gates. Existing benchmarks do not test whether defenders can detect this activity among routine work under a limited review budget. We introduce MOLE, an open benchmark of 150 AI-operated accounts sharing 9 stateful services over 30 workdays, with 12 threats and 8 corpora from four models totaling roughly 20 billion tokens. Of 39 agent models, 72% complete most assigned harmful objectives and agent refusal does not predict completion. MOLE enables comparison of 40 monitors across corpus generators, observability levels, and threats; even the best evaluated monitor in our single-day audit-event comparison misses nearly half of completed harm. MOLE also enables monitor development: benchmark-guided search improves a mid-tier monitor by 49-64%, while selective use of a stronger monitor improves budget-AUC by 10% over applying it to every account-day at comparable modeled cost.