Language models struggle to match expert designs in cancer trial setups
TrialDesignBench: Benchmarking Language Models for Phase-I Trial Design Against Clinical Knowledge
Artificial IntelligenceComputation and Language
Summary
Designing early-phase cancer drug trials requires carefully deciding how to test different doses on patients. The authors created a new test called TrialDesignBench to see how well advanced language AI models can suggest these trial designs based on recent FDA rules. They found that current models don’t perform as well as human experts, partly because the AI struggles to infer hidden factors and follow step-by-step decision rules used by experts. Their benchmark offers a way to better evaluate and improve AI tools for helping design safer and more effective clinical trials.
What this means in practice
- •For clinical trial designers: Assess and improve AI tools aimed at designing statistically valid cancer drug dose-finding trials under current FDA guidelines.
- •For medical ai developers: Use TrialDesignBench as a standardized test to train and evaluate language models on the complexities of early-phase clinical trial design.
Authors
Yihan Hank Tang, Peng Yang, Jiayi Xin, Tianqi Shang, Xuyang Chen, Ying Yuan, Yi Lian, Qi Long
Abstract
AI is used in much of the clinical trial pipeline. However, selecting the statistical design of a trial, a crucial step that determines how drug doses are tested on humans and evaluated, has seen almost no AI progress. LLMs appear to be a natural fit given their strengths in biomedical knowledge and statistical reasoning. Nevertheless, there is currently no appropriate way to evaluate such abilities because recent regulatory and methodological shifts, most notably FDA's Project Optimus, have redefined what constitutes a reasonable design and rendered many previous designs obsolete. To lay the groundwork for AI-assisted trial design, we introduce TrialDesignBench, a benchmark of 617 Phase-I/I-II oncology dose-finding trials, each paired with an expert-validated reference design that is consistent with the latest FDA regulatory initiatives. This reference standard is produced through a formalization of the expert design process. We evaluate five frontier LLMs, none of which matches the reference standard. Our ablation experiments trace this performance gap to two main causes, inference of latent, unrecorded design determinants from trial sources (supplying them raises class-wise balanced accuracy by 0.22) and structured decision-tree reasoning (+0.24). TrialDesignBench offers a reproducible, contemporary-standard objective for evaluating and enhancing AI-assisted trial design.