Pretrained image generators show competence on visual tasks zero shot
Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluation
Computer Vision and Pattern Recognition
Summary
The paper studies how well image-generating AI models can understand and interpret images without additional training, called zero-shot learning. The authors created a new benchmark called ProbeGen to test tasks like depth estimation, segmentation, and counting using text prompts. They found that while specialized AI models perform better on known data and run more efficiently, image generators are often more adaptable when faced with new kinds of images and more skilled at complex reasoning. This suggests that image generators have some surprising abilities in image understanding beyond just creating pictures.
What this means in practice
- •For computer vision engineers: Use image generators to handle novel visual tasks without retraining, benefiting robustness on unfamiliar image types and complex reasoning.
- •For natural language interface developers: Build interfaces that use text prompts to perform multiple visual perception tasks with one generative model, enabling flexible image analysis tools.
Authors
Shangzhe Di, Zhaokai Wang, Weidi Xie
Abstract
Recent work, such as Vision Banana, shows that lightweight instruction tuning can enable an image generator to achieve state-of-the-art performance across multiple visual perception tasks. Motivated by this perspective, we ask how far image generators can go on public visual perception benchmarks in a zero-shot setting. We introduce ProbeGen, a benchmark for zero-shot generative perception that casts monocular depth estimation, referring/reasoning segmentation, and object counting as conditional generation tasks specified through text prompts, and compares 20 models in total---including proprietary and open-weight image generators, specialist perception models, and MLLMs---across 11 published benchmarks. We observe that pretrained image generators show measurable zero-shot perceptual competence, but with a clear trade-off: specialist models remain stronger for in-distribution accuracy and efficiency, while generative models are often more robust under distribution shift and better at compositional semantic reasoning. We hope this study helps establish zero-shot generative perception as a meaningful research direction and provides a useful foundation for future work at the intersection of visual generation and understanding.