CaptchaAgent improves solving accuracy with visual reasoning approach
CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving
Computer Vision and Pattern Recognition
Summary
Modern CAPTCHAs that test computers require detailed understanding of images and logical reasoning, which existing AI models often fail at. The paper presents CaptchaAgent, a new method that first breaks down CAPTCHA images into detailed visual elements before deciding the answer. This step-by-step approach helps the system avoid mistakes and improves its success rate significantly. The authors tested CaptchaAgent on two difficult CAPTCHA datasets and found it outperforms previous methods by a notable margin.
What this means in practice
- •For security software developers: Build more reliable CAPTCHA solvers that verify visual details and reasoning steps to reduce error in automated form submissions.
- •For automated testing teams: Improve browser automation tools by integrating structured visual reasoning to handle complex interactive tasks like CAPTCHA challenges.
Authors
Shangquan Sun, Ruoyu Chen, Zhirui Liu, Xiang Liu, Chenghao Qian, Wenqi Ren, Xiaochun Cao
Abstract
Modern reasoning CAPTCHAs require fine-grained perception, compositional relation reasoning, and grounded browser interaction. End-to-end vision-language model agents remain brittle in this setting because a missed object, incorrect count, or confused relation can propagate directly to the final action. To address this problem, we present CaptchaAgent, a structured visual reasoning framework for interactive CAPTCHA solving. Instead of predicting an action directly from a screenshot, CaptchaAgent first constructs a visual intermediate representation (V-IR) that explicitly records entities, attributes, and spatial or topological relations. A symbol-only ASCII protocol further requires the parser to commit to a concrete object set and count. Building on this global representation, localized evidence refinement revisits ambiguous regions at higher effective resolution, while counterfactual consistency verification tests whether the resulting action remains stable under task-preserving transformations. Across three proprietary backbones on Open CaptchaWorld, CaptchaAgent improves Pass@1 by 21.3-25.8 percentage points over a commercially available browser agent baseline evaluated with the same backbones. On the separate offline Oedipus-data benchmark, it further improves average Pass@1 by 9.2-12.3 percentage points over the Oedipus solver across three matched backbones. The intermediate traces show that exact counting, continuous geometry, and ordered execution remain major sources of automation difficulty for the evaluated agents, providing practical guidance for designing more resilient compositional CAPTCHAs.