MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
2026-08-31 • Artificial Intelligence
Artificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new test called MNIST-PRO to study how AI agents recognize digits when they can only see parts of an image over time, rather than the whole picture at once. They tested different AI models with various memory types to see how well they could remember and piece together these partial glimpses. The authors found that while the models do well when they see everything at once, they struggle when information is limited, often stopping too early or not fixing wrong guesses. This shows the importance of not just seeing pieces but also building and updating a clear mental picture.
AI agentspartial observabilityworking memoryperceptual stateMNIST datasetactive sensingsequential glimpsemultimodal modelsmemory representationbelief revision
Authors
Vernon Toh, Navonil Majumder, Zhengyuan Liu, Nancy F. Chen, Soujanya Poria
Abstract
AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, as agents struggle to integrate fragmented glimpses. Second, agents often stop exploring before they see the full sequence. Third, models often fail to revise early, incorrect beliefs even when faced with subsequent contradictory evidence. These results show that simply acquiring visual evidence is not enough. Agents must also be able to build and update a reliable perceptual state.