Large dataset trains agents to solve interactive captchas accurately
CaptchaArena: A Large-Scale, Fine-Grained Dataset for Training Computer-Use Agents on Interactive CAPTCHAs
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Interactive CAPTCHAs are puzzles designed to stop bots, but computers still struggle to solve many types well. The authors created CaptchaArena, a huge collection of 50,000 different CAPTCHA puzzles with detailed step-by-step solution data. Using this data, they trained an agent called CaptchaAgent that can solve 20 different CAPTCHA types with high accuracy. This work shows that having many examples and detailed computer actions helps improve CAPTCHA-solving programs. Both the dataset and agent are publicly available for further use.
What this means in practice
- •For security software developers: Use CaptchaArena data and CaptchaAgent to improve automated CAPTCHA testing tools ensuring security measures are robust.
- •For automation tool builders: Build advanced interactive agents capable of navigating different CAPTCHA types for automated web interaction tasks.
Authors
Zhenhao Zhang, Zhaoyu Fan, Haohan Ying, Jingwen Hu, Hancen Fan, Junhao Zhou, Zitian Chen, Linchao Zhu
Abstract
Interactive CAPTCHAs remain challenging for computer-use agents, while existing datasets face trade-offs among type coverage, interaction fidelity, and trajectory supervision. To address these gaps, we present CaptchaArena, the first large-scale, fine-grained training dataset for interactive CAPTCHA solving. It contains 50K puzzles across 20 CAPTCHA types and 5 interaction modes, with every solution verified through execution. CaptchaArena provides 50K screenshot-action trajectories, including 46K with step-by-step reasoning annotations. It also includes fine-grained pixel-mask annotations for irregular targets. Using CaptchaArena, we train CaptchaAgent, a single 9B policy for all 20 CAPTCHA types, with supervised fine-tuning followed by reinforcement learning. The environment verifier directly provides the RL reward. Supervised fine-tuning reaches 70.5 Pass@1, and reinforcement learning further improves it to 71.7, while also improving performance on two external benchmarks. These results demonstrate the value of large-scale, fine-grained computer-use supervision for training interactive CAPTCHA agents. We release CaptchaArena and CaptchaAgent at https://github.com/X0X0X00/CaptchaArena.