Multimodal AI struggles with step-by-step jigsaw puzzle help
PuzzleMate: Benchmarking MLLMs for Egocentric Puzzle Assistance
Computer Vision and Pattern Recognition
Summary
Helping people solve puzzles requires very detailed and careful instructions that match what they see and do. The paper studies advanced AI models that can look and talk about images to see how well they can guide someone step-by-step to complete a jigsaw puzzle. The authors found that even the best current AI models still have trouble understanding exactly how pieces fit and what comes next in the right order. They created a way to test these AI models and pointed out several key problems that stop them from being good puzzle helpers.
What this means in practice
- •For smart home device developers: Improve AI assistants to give precise, real-time guidance for assembling household items based on users' current actions.$Commercial implications: Enables creation of AI-enabled devices that provide hands-on stepwise instructions for home assembly tasks, enhancing user experience.
- •For robotics engineers: Design robots that can interpret complex spatial arrangements and provide sequential assembly instructions in human environments.
Authors
Avijit Dasgupta, Shayon Dasgupta, Zakaria Laskar, C. V. Jawahar, Karteek Alahari
Abstract
Personal AI assistants hold the potential to evolve from digital interfaces into embodied companions capable of guiding users through complex physical activities. For these assistants to become integral to daily life, they must do more than identify objects; they must provide precise, step-by-step instructions that align with a user's real-time progress. While Multimodal Large Language Models (MLLMs) show promise in general visual understanding, their ability to deliver grounded, sequential guidance for fine-grained manipulation tasks remains largely unverified. In this paper, we choose the jigsaw puzzle as a strategic testbed for this capability. Unlike general object recognition, puzzle solving demands high-precision spatial reasoning, the ability to distinguish between minute geometric variations, and a rigorous adherence to sequential logic. We investigate this capability through PuzzleMate, a novel framework focused on jigsaw puzzle solving captured through an egocentric viewpoint. We deploy PuzzleMate in a user-in-the-loop study to evaluate how well state-of-the-art MLLMs perceive the current puzzle state and generate actionable next-step instructions. Our analysis reveals seven key bottlenecks that limit their effectiveness. Building on these insights, we propose a benchmark that enables systematic evaluation of MLLMs' reasoning capabilities for puzzle solving. Our findings reveal a substantial performance gap in current models like GPT-5.2 and Gemini-2.5-Pro; while these MLLMs are highly capable, they struggle to navigate the intricate reasoning and sequential logic essential for jigsaw puzzle assistance.