Sciencevision benchmark measures ai understanding of 3d scientific data

ScienceVision: Can AI Observe and Reason Like a Scientist?

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Scientific discovery often depends on understanding 3D structures, like proteins or organs, and using reasoning to interpret them. The authors created ScienceVision, a benchmark to test how well AI can answer expert-level questions about 3D scientific objects from various fields like biology and architecture. They made 250 questions paired with 3D models and tested 12 AI models in different ways, including interacting directly with the 3D data or just 2D images. Their results show AI still struggles, with the best average score only about halfway correct, but models do better when working directly with 3D data than with images.

What this means in practice

  • For medical imaging teams: Evaluate AI tools designed to analyze complex 3D scans by testing their ability to answer detailed domain-specific questions with spatial reasoning.
  • For mechanical design engineers: Test AI assistants on understanding 3D CAD models for complex spatial relationships relevant to mechanical components and assemblies.

Authors

Tianyi Ma, Jing Gu, Haozhe Zhao, Liang Chen, Yunfan Zhang, Haijun Su, Jialong Wu, Le Xue, Arnav Das, Dustin Tran, Yinfei Yang, Andrew Dai, Kuan Li

Abstract

Scientific discovery builds on accurate observations interpreted through domain knowledge and reasoning. For scientific 3D data, this requires understanding structures and spatial relationships, from protein contacts to anatomical organization. Existing multimodal benchmarks largely evaluate text and 2D images, while scientific 3D evaluations vary in representations and protocols across disciplines, limiting cross-disciplinary comparison. To assess these foundational capabilities, we introduce ScienceVision, a cross-domain benchmark for scientific 3D understanding and reasoning through expert-level visual question answering (VQA). We construct 250 questions, with 50 each from Biology, Chemistry, Medical Imaging, Mechanical CAD, and Architecture, through a pipeline combining synthetic question generation with domain-expert review and annotation. Each task pairs a native 3D artifact with a scientific question, a specified response format, and a task-specific reference answer, requiring domain knowledge to be applied to geometric or topological evidence. We evaluate 12 models on the same questions in three settings: visual question answering with native 3D artifacts accessed through an agent harness's file and code-execution tools; the same workflow augmented with expert-inspired viewer inspection and measurement tools exposed through the Model Context Protocol (MCP); and visual question answering from prescribed rendered 2D images using an agent harness. All settings use the same task-specific deterministic final-answer scoring. The inspection environment supports view changes, target isolation, and slice inspection. In this main comparison, the best mean answer score is 0.53 out of 1. Mean scores in the native-artifact workflow without additional MCP tools exceed those in the rendered-image workflow for every model, by 17.2 percentage points on average.