Perception test challenge expands to city-scale audio visual reasoning
Perception Test 2026: Challenge Summary and Extension to City-scale Audio-Visual Reasoning
Computer Vision and Pattern RecognitionMachine Learning
Summary
Figuring out how machines understand complex sounds and videos over a big city is hard. The Perception Test 2026 challenge added new tests using videos from city-scale walking tours to explore this. The authors showed that while some expensive, multi-step computer setups can solve these tough spatial and multi-sensory questions, standalone models struggle more. They also introduced a versatile model that worked okay across all types of tests. This helps understand what kinds of computer intelligence handle real-world, large-scale audio and video puzzles best.
What this means in practice
- •For augmented reality developers: Enhance AR experiences with improved spatial and audio-visual reasoning from city-scale video and sound data.
- •For urban planning teams: Use multimodal city-scale video reasoning to better understand and simulate urban environments for planning.
Authors
Fedor Kitashov, João Carreira, Shiry Ginosar, Dima Damen, Andrew Zisserman, Viorica Pătrăucean
Abstract
Continuing the Perception Test challenge series, we organised the fourth edition as a workshop at the European Conference on Computer Vision (ECCV) 2026 in Malmö, Sweden. This edition focused on spatial intelligence and featured four different tracks: unified multiple-choice videoQA and grounded videoQA from the original Perception Test benchmark, alongside two new tracks based on city-scale walking-tour videos (KilometerAudio and KilometerVision). In this report, we describe the new benchmarks used for the city-scale tracks and summarise the winning solutions across all tracks, including a generalist model that competed across all tracks with satisfactory performance. The winning solutions in the newly added city-scale tracks demonstrated that complex spatial and multimodal reasoning can be solved by expensive agentic pipelines, but remains difficult for multimodal models used standalone.