Ai agents tested for finding mobile app security flaws with executable checks
MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes
Cryptography and SecurityArtificial IntelligenceMachine Learning
Summary
Finding security problems in mobile apps is slow and often requires a lot of work to understand the details. The authors created a new way to check if reported problems really exist by running special tests called probes that see which security rules were broken. They built a tool called MobileCybench with these probes to test AI programs on Android apps. The best AI found security problems in over half the apps when it pretended to be a malicious app. This method also found many new problems that app makers confirmed were real.
What this means in practice
- •For mobile app security teams: Test AI-driven attack simulations on Android apps to quickly identify security weaknesses using executable probes.
- •For cybersecurity analysts: Use executable security probes to validate and classify AI-reported vulnerabilities for faster, more precise threat assessment.
Authors
Andy K. Zhang, Ava Huang, Joey Ji, Wai Han, Thomas Qin, Nardos Demilew, Michael Tian-Yue Liu, Brian Song, Riya Dulepet, Brian Wang, Kyleen Liao, Cuiyuanxiu Chen, Nishka Kacheria, Andrew Wu, Pratham Rangwala, Xinjie Wang, Laura Gomezjurado Gonzalez, Anita Ding, Benjamin Yi, Daniel E. Ho, Dan Boneh, Dawn Song, Ion Stoica, Percy Liang
Abstract
AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application, and require considerable human labor to process. To mitigate this, we introduce a framework for evaluating vulnerability reports via probes, executable checks of security properties. A reported exploit is evaluated by replaying it against the application and running the probes: a triggered probe indicates both that the exploit succeeded and which security property it violated. As a probe encodes a security property rather than a known vulnerability, it can detect vulnerabilities that were not known when the probe was written. We instantiate the framework as MobileCybench, a benchmark for vulnerability discovery by AI agents in 13 Android applications, with 495 probes written and reviewed by the authors. We evaluate 5 coding agents (OpenCode with GPT-5.5, GPT-5.6-Sol, and GLM-5.2; Claude Code with Opus 4.8 and Opus 5) under 4 settings: as a malicious app on the victim's device or as a remote attacker with a low-privilege account, each with either only an obfuscated APK or access to the application's source code. Given only the obfuscated APK, the top agent, OpenCode with GPT-5.6-Sol, triggers probes in 53.8% of applications in the malicious-app setting and 16.7% in the remote-attacker setting. With source code, the trigger rate across all agents and both attack settings increases from 28.8% to 32.8%. Building and running the benchmark surfaced 23 previously unreported vulnerabilities, the majority of which have been confirmed by maintainers.