Text to image models tested on hardest visual challenges
GenOlympiad: Benchmarking Frontier Image Models at Their Capability Boundaries
Computer Vision and Pattern Recognition
Summary
Text-to-image AI models can create impressive pictures from words, but it’s unclear where they struggle most. The authors created GenOlympiad, a tough test with 9 types of difficult challenges, like counting objects or following rules exactly, to see how well these models do. They tested over 20 models and found even the best could only get about 24% perfect on hard prompts. This work helps show the limits of current image generation AI and gives tools to measure progress clearly.
What this means in practice
- •For ai model developers: Evaluate and improve text-to-image models by benchmarking against the most challenging visual reasoning tasks with clear, objective metrics.
- •For creative ai tool builders: Use the GenOlympiad benchmark to identify and reduce common failure modes in image generation for complex or precise user prompts.
Authors
Xi Hu, Martin Bell, Liang Zheng
Abstract
Text-to-image models have made tremendous progress and can produce high-quality images. However, a question which would be asked in any other discipline is where the frontier models fall short. We introduce GenOlympiad, a benchmark which provides the most challenging text prompts from nine capability axes for stress-testing frontier text-to-image models. The assessed capabilities include generating large quantities, performing deep reasoning, ensuring hard physical consistency, following complex rules and so on. The prompts are designed such that prompt alignment is visually verifiable (e.g., counting) instead of relying on subjectiveness (e.g., style). Based on this, we develop an automated evaluation framework which decomposes the prompt into easy and hard questions, and for most hard questions, draws help lines to ground evidence for accurate grading. We evaluate 20+ strong text-to-image models, including Nano Banana Pro, GPT-Image-2.5, Seedream-5.0 and so on. We report that even the best-performing T2I model (Nano Banana Pro) only achieves a strict score of 23.9 out of 100 and that coding models such as GPT-6 Astra have higher prompt alignment scores (but not as visually pleasing). We also derive insights in model evolution, alignment with Elo scores, and effectiveness of our grounding-based evaluation protocol. Code is available at https://github.com/GenOlympiad/GenOlympiad and the project page is at https://genolympiad.github.io/