Summary
People want to know if tools that create images from text can also understand those images well without extra training. To explore this, the authors tested 20 different models, including image generators and specialized perception systems, on various vision tasks like depth estimation and object counting. They found that while image generators have some ability to understand images right away, specialized models still perform better when tested on data similar to what they were trained on. However, image generators tend to handle new or unexpected situations better and can reason about complex scenes in a more flexible way. This work suggests a new path for combining image creation and understanding in future research.
image generatorzero-shot learningvisual perceptiondepth estimationsegmentationobject countingdistribution shiftcompositional reasoninginstruction tuningmultimodal large language models
Authors
Shangzhe Di, Zhaokai Wang, Weidi Xie
Abstract
Recent work, such as Vision Banana, shows that lightweight instruction tuning can enable an image generator to achieve state-of-the-art performance across multiple visual perception tasks. Motivated by this perspective, we ask how far image generators can go on public visual perception benchmarks in a zero-shot setting. We introduce ProbeGen, a benchmark for zero-shot generative perception that casts monocular depth estimation, referring/reasoning segmentation, and object counting as conditional generation tasks specified through text prompts, and compares 20 models in total---including proprietary and open-weight image generators, specialist perception models, and MLLMs---across 11 published benchmarks. We observe that pretrained image generators show measurable zero-shot perceptual competence, but with a clear trade-off: specialist models remain stronger for in-distribution accuracy and efficiency, while generative models are often more robust under distribution shift and better at compositional semantic reasoning. We hope this study helps establish zero-shot generative perception as a meaningful research direction and provides a useful foundation for future work at the intersection of visual generation and understanding.