Summary
Robotic surgery relies on cameras called endoscopes to see inside the body, but previous datasets only used one camera view at a time, limiting how well computer systems can understand surgical scenes from new angles. The authors created iMED, a large collection of synchronized video from four cameras placed independently during surgeries on different anatomical areas, enabling better testing of 3D understanding methods. They found many current techniques struggle when tested on viewpoints not used during training, but models that use geometric information or large-scale pretrained models handle this better. This new dataset helps improve and measure how well surgical vision systems can generalize to unseen perspectives, a key step toward safer and more accurate robot-assisted surgeries.
What this means in practice
- •For clinical ai developers: Train and evaluate surgical vision AI to generalize better by using synchronized multi-camera data from different anatomical sites.
- •For robotics software teams: Improve robot-assisted surgery navigation and manipulation by benchmarking algorithms with new multi-endoscope 3D perception data.
Authors
Sierra Bonilla, Fengyi Jiang, Chinedu Nwoye, Jingpei Lu, Kailey Reardon, Humphrey W. Chow, Francisco Vasconcelos, Sophia Bano, Adam Schmidt, Omid Mohareri
Abstract
We introduce iMED, the first synchronized multi-endoscope dataset for robot-assisted minimally invasive surgery (RAMIS), designed to address a fundamental limitation in surgical vision benchmarking: the absence of independent held-out viewpoints. Existing surgical datasets capture single-trajectory sequences from a single endoscope, making it impossible to distinguish geometric generalization from photometric interpolation along a narrow forward-facing path. iMED provides 340 sequences ~170K synchronized timepoints with 4 views per timepoint) recorded simultaneously from two independent stereo endoscopes across ex vivo, postmortem, and live surgical settings, spanning 14 specimens with diverse anatomical regions and motion regimes. The dataset includes calibrated camera intrinsics, ArUco-based frame-wise pose estimates with uncertainty quantification, instrument segmentation masks, and rich clinical metadata. Using a train-on-one-endoscope, test-on-another evaluation protocol, we benchmark 23 state-of-the-art methods across rigid and deformable novel view synthesis, pose estimation, feature matching, and monocular depth estimation. Our experiments show that methods relying primarily on photometric supervision degrade substantially under this held-out-endoscope setting, while methods with explicit geometric regularization are more robust across views in our benchmark. Notably, large-scale foundation models transfer surprisingly well to surgical imagery, often outperforming domain-specific models, suggesting pitfalls in current fine-tuning protocols. iMED establishes a new evaluation protocol for geometric generalization in surgical vision.