Sciencebuddy-pv audits imaging model for vascular intervention research
ScienceBuddy-PV: A Verifier-Governed Recursive Improvement Protocol for Panvascular Intervention Research
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Panvascular interventions are medical procedures on blood vessels. The authors adapted an AI system called ScienceBuddy to help analyze imaging data for these procedures by creating a special protocol to verify parts of the AI’s performance. They tested a specific model on images showing vessel narrowing and measured how well it marked problem areas, including its ability to say when it was unsure. This work provides a way to measure and reproduce support and hesitation in AI models, but does not claim it improves itself or directly helps patients yet.
What this means in practice
- •For medical image analysts: Assess and audit image segmentation models for blood vessel narrowing with measurable criteria including support and abstention rates.
- •For clinical software developers: Implement reproducible protocols to verify AI model components in vascular imaging to improve transparency before clinical deployment.
Authors
Lingsen You, Yujun Guo, Xinyu Zhong, Zijie Feng, Wentong Wang, Li Shen, Junbo Ge
Abstract
ScienceBuddy-PV adapts the ScienceBuddy recursive-in-recursive design to panvascular intervention research through a structured evidence ledger, device-vessel ontology, abstention contract and retained failure bank. Recursive harness optimization and outer-loop model learning, including reinforcement learning, are specified but not executed. We audit a fixed component, rather than the complete agent system, using separate ARCADE syntax and stenosis tasks (1,000 training, 200 validation and 300 test images per task). A shared task-masked convolutional model was trained once per representation setting; its parameters and the heuristic support gate were then held fixed, with the stenosis threshold selected on validation data. On the locked test sets, syntax Dice was 0.475 (95% CI 0.459-0.490) and stenosis Dice was 0.231 (95% CI 0.214-0.249). Stenosis pixel AUROC was 0.957 and AUPRC was 0.180 at prevalence 0.00975. The gate accepted 76.7% of stenosis-frame outputs versus 100% for a score-only comparator. In a constructed mask audit, it accepted 44.3% of reference candidates and rejected 55.7%, exposing the cost of abstention. Acceptance does not establish correctness or clinical safety. This public-data component audit makes support and abstention measurable and provides a reproducible protocol for future collaborative agent studies. It does not demonstrate recursive improvement, patient-level validity or clinical device recommendations.