Screwtape edits transcripts to bypass AI action monitors
Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming
Cryptography and SecurityArtificial Intelligence
Summary
AI systems are often monitored to prevent harmful actions, but sometimes these monitors can be tricked. The authors created Screwtape, a tool that edits recorded AI agent actions to produce harmful behaviors that the monitors fail to detect. Screwtape uses another AI to check that the edited harmful actions make sense and are severe enough before trying again if the monitor catches them. This approach found real blind spots in existing monitors, showing how attackers can bypass defenses without direct access to a system. Developers can use these findings to improve their monitoring systems.
What this means in practice
- •For ai safety teams: Identify blind spots in AI action monitors by generating harmful cases that evade detection, helping to improve system robustness.
- •For software security engineers: Use edited AI transcripts to uncover security weaknesses in action monitoring systems without needing direct access to the protected infrastructure.
Authors
Gosha Kozhevnikov, Tyler Tracy
Abstract
We present Screwtape, a pipeline that converts real agent transcripts into attacks that evade action-only monitors. An LLM attacker edits a transcript so that the agent causes a specified harm, such as leaking an API key or spawning an unmonitored agent. The edited actions are never executed, so we use an LLM verifier to reject attacks that are inconsistent with the original transcript or below a severity bar. The attacker revises the edit until the monitor no longer flags it. We applied Screwtape to 94 Claude Code and Codex transcripts from Redwood Research staff, optimizing against our own monitors. Across 252 runs, it produced 35 attacks that passed the verifier and evaded these monitors. Six of them, confirmed valid by human review, also evaded a held-out monitor provided by a frontier developer. We therefore demonstrate that a red-teamer without access to a developer's infrastructure can find confirmed failures of the developer's monitor. We expect to find more attacks when the pipeline runs inside the developer's infrastructure, with frontier attacker models that lack refusal safeguards and direct optimization against the production monitor. Developers can use these attacks to fix monitor blind spots, and evaluators can use them as counterexamples to safety claims about the monitor.