Vision models learn to explore beyond screens for better perception
Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
People usually train computers to recognize things they see on screens, but they often just passively look at images without exploring. The authors trained a model to actively decide where to focus beyond just the screen area, improving its observation skills. They first taught it basic visual inspection by showing examples from user interfaces, then refined it with trial and error to better explore useful parts of images. This approach lets the model use what it learned from screens to improve understanding of other types of visual content too.
What this means in practice
- •For mobile app developers: Build apps that help AI better understand screen content by teaching models to look beyond just visible GUI elements.
- •For security monitoring teams: Improve AI systems for surveillance by enabling models to actively decide where to inspect beyond standard camera views.
Authors
Ruilin Yao, Xinyu Qiu, Tianyu Zou, Yi Rong, Shengwu Xiong, Shuheng Shen, Changhua Meng
Abstract
Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at https://github.com/inclusionAI/UI-Venus/tree/LBS and https://huggingface.co/inclusionAI/LBS-27B, respectively.