Summary
Smartphone apps that use artificial intelligence to read and act on screenshots can accidentally reveal private information or figure out personal user details. The authors have created PriMobiBench, a new test framework to measure these privacy risks systematically. They built a dataset from popular apps showing many examples where sensitive data can be leaked or users profiled visually. Their experiments show AI models can often correctly extract private information and guess user profiles, but masking irrelevant private parts of the screen first can reduce this risk while mostly keeping the app’s performance. This work helps show how screenshot-based AI can threaten privacy and points to ways to protect users.
What this means in practice
- •For mobile app developers: Evaluate and reduce privacy risks in apps that automate smartphone tasks by interpreting screenshots with AI.
- •For cloud service operators: Enhance privacy protections by integrating masking techniques that block irrelevant sensitive UI elements before AI processing.
Authors
Qihang Cen, Tianshuo Cong, Da Song, Xinlei He, Jiaxing Song, Ke Xu, Qi Li
Abstract
Mobile GUI agents increasingly rely on Vision-Language Models (VLMs) to automate smartphone tasks by interpreting screenshot streams. However, this design introduces serious and underexplored privacy risks, including direct leakage of sensitive on-screen information and unintended user profiling. The absence of standardized benchmarks makes it difficult to quantify these risks in realistic mobile agent workflows. To address this gap, we propose PriMobiBench, the first benchmark for systematically evaluating privacy leakage and visual profiling in screenshot-driven mobile agents. It provides a unified pipeline for data generation, agent trajectory construction, and multi-model evaluation. We also introduce MobiLeak, a dataset of execution traces from 16 apps, covering 25 privacy attributes with 2,960 embedded privacy instances. Our results reveal substantial risks: (1) VLMs can directly extract sensitive information with up to 82.5% success rate; (2) beyond explicit leakage, they can infer user profiles from aggregated visual evidence with approximately 70% success. We further propose a mitigation that masks privacy-sensitive but task-irrelevant UI elements before cloud processing, reducing profiling success by up to 58% with only approximately 8% performance loss. Overall, our work provides the first systematic benchmark for visual privacy risks in mobile GUI agents, demonstrates that both leakage and profiling are feasible at a highly concerning level, and offers a practical direction for mitigation.