Summary
MRI scans help scientists understand brain aging and disease, but different processing methods can lead to different conclusions even with similar measurements. The authors created a way to check how reliable claims are when changing methods or data sources by turning brain scan details into easy-to-check summaries called claim cards. They tested this on thousands of scans and found that some claims are stable while others are fragile or depend on specific methods. Specialists and AI tools reviewed these claims for clarity and trustworthiness. This framework helps scientists and medical experts see how trustworthy brain study results are without needing a perfect standard to compare to.
What this means in practice
- •For clinical imaging teams: Audit MRI processing pipelines to identify trustworthy brain measurement claims across datasets and methods for aging and disease analysis.
- •For medical ai developers: Develop more reliable diagnostic tools by incorporating claim stability assessments for MRI brain morphometry features across different processing approaches.
Authors
Duy-Cat Can, Dang-Duat Tran, Anh-Khang Nguyen, Quoc-Bao Tran, Trung-Hieu Do, Van-Tu Nguyen, Duy-Thanh Vu, Oliver Y. Chén, Hoang-Quynh Le
Abstract
MRI morphometry pipelines are often evaluated by measurement agreement or downstream accuracy, yet their scientific value depends on the conclusions they support. Two pipelines can produce similar feature-level results yet support different conclusions about aging, cognitive decline, or neurodegeneration. We introduce Scientific Claim Stability (SCS), a claim-centric evaluation framework that converts region-of-interest (ROI) morphometry tables into auditable claim cards and evaluates their support under changes in dataset, processing pipeline, atlas, feature family, and target. Across 10,370 MRI scans from three datasets and five cohorts, SCS generates 2,783 claim cards and evaluates 89 claim-family pairs. The evaluation distinguishes stable, transferable, and fragile claims. FreeSurfer 8-FastSurfer comparisons show high evidence concordance and low anatomical-support overlap under the primary covariate specification. Acquisition-adjusted analyses show that the magnitude of support disagreement depends on the model specification. Structured review by specialists in geriatrics, neuroscience, and radiology, alongside auxiliary ChatGPT and Gemini reviewers, assesses claim plausibility, clarity, faithfulness, usefulness, and caution. SCS provides a reproducible way to audit the evidential stability of morphometry claims without assuming a universal gold standard.