Image generation boosts flexible visual reasoning in AI models
Reasoning with Image Generation
Computer Vision and Pattern Recognition
Summary
Many AI systems can solve problems by breaking them into smaller steps using text alone, but struggle with tasks that need visual thinking. The authors created a new method called ReImaGin that lets AI generate and change images based on instructions in natural language. This method helps AI better understand and solve difficult visual puzzles like predicting collisions or combining views of a room. It works better than only using text or special vision tools because it can create many kinds of pictures on demand.
What this means in practice
- •For robotics engineers: Use flexible image generation to help robots visualize and predict spatial interactions beyond fixed sensor outputs.
- •For architectural visualization teams: Generate floorplans and composite room views from multiple inputs to assist design and planning workflows.
- •For video game developers: Create dynamic scene reasoning and collision prediction by integrating image-based reasoning into game AI.$Commercial implications: Enables advanced in-game AI that can flexibly generate and manipulate visual elements, enhancing gameplay realism and interaction.
Authors
Nishad Singhi, Hector Garcia Rodriguez, Aditya Arora, Marcus Rohrbach, Anna Rohrbach
Abstract
Chain-of-thought reasoning has revolutionized natural language processing by enabling large language models (LLMs) to decompose problems into intermediate steps before answering. Yet confining reasoning to the textual domain presents limitations for tasks requiring direct manipulation of visual representations. Recent efforts augment multimodal LLMs with external visual expert tools such as depth estimation or object detection modules, but these remain fundamentally limited by their reliance on narrow, rigid operations that cannot flexibly generate or transform visual content. We propose ReImaGin, which leverages image generation models as a flexible visual reasoning mechanism for multimodal LLMs: unlike fixed-function tools, they accept natural language commands and can perform open-ended visual operations, like removing an occlusion or generating a floorplan from multiple disjoint views of a room. Across six diverse visual reasoning tasks including multi-view spatial reasoning and collision prediction, ReImaGin consistently outperforms both text-only reasoning and specialist vision-tool baselines, with gains of up to 25\%, demonstrating the advantage of flexible, generative visual reasoning.