MintAct combines vision and language to control digital environments
MintAct: A Unified Visual Agent for Digital Environments
Computer Vision and Pattern Recognition
Summary
Interacting with apps on your phone, computer, or web browser involves many different tasks like clicking buttons, navigating through menus, or using on-screen tools. The team behind MintAct created a single smart model that can understand what it sees and uses language to perform all these tasks across different devices and platforms. They trained this model on lots of examples and used smart methods to keep it learning well even when the data or feedback is noisy. MintAct matches or exceeds specialized systems at these tasks, showing it can handle complex digital interactions in one unified way.
What this means in practice
- •For mobile app developers: Enable apps to understand and act on complex user commands involving visual interface elements across Android and iOS with a single integrated model.
- •For enterprise software teams: Automate workflows spanning desktop and web applications by linking visual cues to actions through a unified AI assistant.
Authors
Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy, Kaixin Ma, Vishwesh Kirthivasan, Oğuzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan
Abstract
We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and 8B scales. Through careful design of our environments, data, and training recipes, MintAct models match the performance of per-domain specialists across all of these capabilities. To enable this, we develop a scalable environment and reinforcement learning (RL) infrastructure. On the environment side, we host hundreds of concurrent instances across heterogeneous per-domain backends, serving both trajectory data collection and online RL. To enable efficient and scalable RL training, an asynchronous framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Experimental results show that MintAct achieves state-of-the-art performance (48.9 on OSWorld-Verified) across a wide range of benchmarks at comparable model sizes.