Papers for

clinical trial designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Language models struggle to match expert designs in cancer trial setups

TrialDesignBench: Benchmarking Language Models for Phase-I Trial Design Against Clinical Knowledge

Abstract: AI is used in much of the clinical trial pipeline. However, selecting the statistical design of a trial, a crucial step that determines how drug doses are tested on humans and evaluated, has seen almost no AI progress. LLMs appear to be a natural fit given their strengths in biomedical knowledge and statistical reasoning. Nevertheless, there is currently no appropriate way to evaluate such abilities because recent regulatory and methodological shifts, most notably FDA's Project Optimus, have redefined what constitutes a reasonable design and rendered many previous designs obsolete. To lay the groundwork for AI-assisted trial design, we introduce TrialDesignBench, a benchmark of 617 Phase-I/I-II oncology dose-finding trials, each paired with an expert-validated reference design that is consistent with the latest FDA regulatory initiatives. This reference standard is produced through a formalization of the expert design process. We evaluate five frontier LLMs, none of which matches the reference standard. Our ablation experiments trace this performance gap to two main causes, inference of latent, unrecorded design determinants from trial sources (supplying them raises class-wise balanced accuracy by 0.22) and structured decision-tree reasoning (+0.24). TrialDesignBench offers a reproducible, contemporary-standard objective for evaluating and enhancing AI-assisted trial design.

Mon 9 NovArtificial IntelligenceComputation and Language
The gist
Designing early-phase cancer drug trials requires carefully deciding how to test different doses on patients. The authors created a new test called TrialDesignBench to see how well advanced language AI models can suggest these trial designs based on recent FDA rules. They found that current models don’t perform as well as human experts, partly because the AI struggles to infer hidden factors and follow step-by-step decision rules used by experts. Their benchmark offers a way to better evaluate and improve AI tools for helping design safer and more effective clinical trials.
Open → 2611.12640v1

Patient world models struggle to predict drug effects in trials

Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials

Abstract: Patient world models are increasingly proposed as simulators of clinical trials, and recent surveys rank their capabilities on a ladder from temporal prediction to counterfactual rollouts. Position on that ladder is currently assigned by reading papers rather than by measurement. We turn the counterfactual rung into a measured quantity. From 30,119 completed phase 2/3 trials with results posted to ClinicalTrials.gov we construct 9,996 audited within-trial contrasts from 7,207 trials, and we score a simulator on how much of its accuracy depends on knowing which intervention was tested. The instrument is an intervention-blind reference: the same model, trained on the same trials, with every intervention-identifying field withheld. Interventional skill is the error the full model removes relative to that reference. On a temporal split (878 test contrasts from 655 trials) a registry-trained gradient-boosting simulator removes a fifth of the error of a trivial baseline, but its interventional skill is -0.12 (95% CI -0.21 to -0.05): withholding the intervention makes it better, and no alternative featurisation, stratum, temporal cut or same-molecule retrieval yields an interval above zero. A frontier language model reading only the registered protocol has interventional skill 0.20 [0.08, 0.31]; smaller and open-weight models sit at the limit of detection. Perturbation tests show the large model uses the intervention as such: swapping the arms reverses the sign of its prediction in 86% of contrasts. The blind reference is itself not drug-blind (a language model recovers the withheld drug class 45% of the time from the blind prompt), so interventional skill is a conservative lower bound. A reliability layer shows that ranking by predicted error lowers it but that calibrated error control does not transfer under temporal shift. Benchmark, code, prompts and per-call model outputs are released.

Wed 4 NovArtificial IntelligenceMachine Learning
The gist
Simulating how drugs work in clinical trials is important for research, but it’s hard to tell if these simulations really understand the drug’s effect or just guess from other information. The authors tested a model’s skill at knowing what a drug does by hiding which drug was given and seeing if the model’s accuracy dropped. Surprisingly, some models did better without knowing the drug, meaning they didn’t truly grasp the drug’s effect. Larger language models showed some ability to use the drug information correctly, but overall, it’s clear that many simulators may not know as much about drug actions as we hoped.
Open → 2611.05907v1

Transformer model guides safer dose choices in early drug trials

Transformer-Based Reinforcement Learning for Phase I Dose Allocation

Abstract: Phase I dose-finding trials must learn from sparse toxicity observations while limiting exposure to overly toxic doses. We develop a single reinforcement-learning policy for trials with different numbers of ordered dose levels. Each dose is represented as a token containing its normalized position, accumulated patient count, observed dose-limiting toxicities (DLTs), and current-dose indicator. A Transformer encoder summarizes the dose tokens, trial-conditioned cross-attention produces a fixed-dimensional trial representation, and a three-action policy chooses de-escalation, staying, or escalation. The final maximum tolerated dose (MTD) decision is deliberately kept outside the learned policy: after the planned sample size, weighted isotonic regression and a prespecified safety cap select the dose closest to the target toxicity probability. In the simulations, toxicity-penalized policies reduce DLT burden while unpenalized policies improve selection accuracy at the cost of greater toxicity. These results demonstrate a controllable safety--accuracy trade-off but do not establish superiority over conventional dose-finding designs.

Fri 30 OctMachine Learning
The gist
Finding the right drug dose in early trials is tricky because giving too much can be harmful, and giving too little might not work. The authors created a computer program that learns how to pick doses by looking at past patient reactions and the number of patients tried at each dose. This program uses a special AI model called a Transformer to understand the whole trial situation and then decides whether to increase, keep, or lower the dose. The final decision on the best dose is made using another method to make sure safety is prioritized. Their tests show you can balance being safe and choosing the correct dose, though it doesn't necessarily work better than current methods.
Open → 2611.00395v1

Automated system improves clinical trial eligibility criteria using textual evidence

RECITE: Revising Eligibility Criteria Incorporating Textual Evidence

Abstract: Overly restrictive eligibility criteria contribute to widespread enrollment failure in clinical trials, yet published recommendations for how specific criteria should be broadened remain disconnected from trial design at scale. We introduce RECITE (Revising Eligibility Criteria Incorporating Textual Evidence), a framework for automated discovery, extraction, and implementation of evidence-based eligibility criteria revisions. RECITE uses protocol amendment documents from ClinicalTrials.gov to construct a dataset of 11,913 curated instances across 5,735 trials, each pairing original criteria with documentary evidence and target revisions. We benchmark fifteen LLM configurations under truncated-to-context-window and retrieval-augmented generation conditions, on a 3,116-instance sample under an LLM-as-judge protocol validated against ten domain experts (judge vs. human quadratic-weighted Cohen's $κ_{qw} = 0.694 \pm 0.096$). The best open-weight model, Gemma 2 9B, achieves $κ_{qw}$-aligned ordinal score $3.0 \pm 0.9$ on a 0--4 scale (normalized 0.742) and the best closed-weight model, GPT-4o, reaches $3.0 \pm 0.8$ (normalized 0.745). Using an agentic literature discovery system, we then identified 44 paper--trial matches across five publications providing trial-design and EC-modification evidence. Our findings suggest that existing clinical knowledge can be operationalized into actionable trial design recommendations.

Fri 30 OctArtificial Intelligence
The gist
Many clinical trials fail because their rules for who can join are too strict. The authors created a system called RECITE that uses past trial changes and documents to suggest better, evidence-backed rule adjustments. They tested how well large language models could suggest improvements and found some models perform close to expert humans. This approach helps turn medical knowledge into practical trial design advice.
Open → 2610.41368v1

Language models struggle to judge scientific evidence but improve with training

Do Language Models Know What Compelling Scientific Evidence Looks Like?

Abstract: Expert scientists excel at weighing empirical findings from multiple sources that vary in both statistical reliability and epistemic relevance. To what extent are language models, which are increasingly driving autonomous discovery systems, capable of such evidence deliberation? We introduce EDB-Bio, a scientific reasoning benchmark curated from real-world data in three biological domains: drug-perturbation transcriptomics, CRISPR screens, and human genome-wide association studies (GWAS). Each of EDB-Bio's 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. Throughout, quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. Our work extends calibration training and uncertainty analysis to a realistic scientific forecasting setting with applications to human cancer biology. We release all code, data, and trained models.

Thu 29 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
It can be hard for computers to tell how strong scientific evidence is when studies have different levels of reliability and relevance. This paper created a big test using biology experiments to see if language models can predict how likely a scientific claim is true based on multiple results. At first, the models were often too cautious or unsure and didn’t do much better than guessing. But after special training to better understand uncertainty and make predictions, their skills improved a lot and they could pick experiments more efficiently. The authors also studied how these models update their beliefs and compare to expert scientists.
Open → 2610.41015v1

Algorithm improves best choice finding under budget limits

SHPRR: An Algorithm for Pure Exploration under Constrained Regret

Abstract: Best Arm Identification (BAI) conventionally measures the cost of experimentation by the number of samples collected. In many operational settings, however, the cost of exploration relative to a trusted baseline already in place is equally important. We study BAI under both a fixed sampling horizon and a regret budget. Unlike generic resources considered in constrained pure exploration, regret has a distinctive structure: sampling better arms is less costly relative to the baseline. We theoretically show that the regret cost of exploration decreases across the phases of Sequential Halving, a widely used pure-exploration algorithm. This observation yields improved guarantees on the probability of identifying the best arm and motivates a more effective allocation of the regret budget. We propose Sequential Halving with Proportional Regret Rationing (SHPRR), which front-loads the regret budget to allow more exploration when many arms remain. Experiments on synthetic and real-world instances show that SHPRR improves best-arm identification performance over uniform regret allocation and existing baselines.

Thu 29 OctMachine LearningArtificial Intelligence
The gist
Sometimes it's important to find the best option in a group quickly while keeping the cost of trying options low compared to a trusted choice. The authors studied how to balance exploring many options without exceeding a set regret budget, which accounts for how much worse exploration is compared to the trusted baseline. They improved an existing method by assigning more 'regret budget' early on, allowing better identification of the best option. Tests on both fake and real problems showed this new method works better than others.
Open → 2610.39837v1

Control groups matter in immersive virtual reality therapy trials

Virtual Placebo: a scoping review of control conditions in randomized controlled trials of psychological immersive virtual reality interventions

Abstract: Research on immersive virtual reality (IVR) as a therapeutic tool is growing rapidly, with randomized controlled trials (RCTs) now commonplace. Yet the control conditions in many of these studies that claim strong specific IVR effects vary from usual care to active immersive experiences, making it difficult to draw clinically meaningful conclusions. We conducted a scoping review of recent literature on psychological IVR interventions to assess the main forms of control conditions used and their relations to study outcomes. In this paper, we make the case that most of the recent RCTs studying the effects of specific IVR interventions compare active immersive VR to usual care with no multimedia component. This can lead to strong yet potentially misleading study outcomes and can potentially inflate the IVR interventions' effect. Namely, the lack of an appropriate immersive control condition may risk misattributing general effects of VR devices to the specific content of the intervention. In this review, we discuss the implications of these design choices and propose strategies for including a suitable "virtual placebo" condition to improve the methodological rigor and interpretability of results.

Fri 23 OctHuman-Computer Interaction
The gist
Studies are testing virtual reality (VR) to help with psychological treatments. This paper found many tests compare VR therapy to usual treatments without using a similar VR experience as a control. The authors warn that this can make VR look more effective than it really is. They suggest using a 'virtual placebo' control in future studies to better understand what parts of VR treatments actually work.
Open → 2610.32232v1

Bayesian method optimizes decision time with error and abstention limits

Bayesian Fixed-Confidence Best-Arm Identification with Abstention

Abstract: We study the Bayesian fixed-confidence best-arm identification problem in which a learner can abstain from making a terminal recommendation. Subject to a target error probability $δ$ and an abstention budget $α$, we propose Screen-and-Confirm (S&C) and analyze its expected stopping time. We further derive the minimum expected stopping time for any algorithm that satisfies these constraints. For Gaussian priors and rewards, as $(δ,α)\downarrow(0,0)$, we establish matching upper and lower bounds up to multiplicative constants, with rate $κ_ν^2[1+\log((α+δ)/δ)]/(α+δ)$, where $κ_ν$ is the prior density of the top-two gap at zero, highlighting that nearly tied instances govern the sampling cost. When abstention is disallowed ($α=0$), S&C yields the optimal $Θ(1/δ)$ dependence for fixed model parameters, removing the additional $\log(1/δ)$ factor in the upper bound of Jang et al. (2024) under our Gaussian assumptions. We further characterize the exact multiplicative constant of the minimum expected stopping time for practical regimes including (i) fixed $α\in (0,1)$ and (ii) $α/δ\to\infty$, which are also achieved by S&C.

Wed 21 OctMachine Learning
The gist
The paper looks at how to quickly and confidently find the best option among many uncertain choices while allowing the option to opt out sometimes. The authors introduce a new approach called Screen-and-Confirm (S&C) and analyze how long it takes to decide on the best choice given limits on error and how often one can abstain. They find exact formulas for the minimal time any method needs to meet these limits, especially when the differences between options are small and uncertain. Their method improves on previous work by removing extra delays in decision time when no abstention is allowed.
Open → 2610.30384v1

Simple method improves picking best items from noisy choices

A Simple and General Approach to top-$k$ Selection

Abstract: This paper considers the Bayesian top-$k$ identification problem, wherein a decision maker is tasked with selecting $k$ items from a pool of items of unknown quality. To assess the items, the decision maker adaptively allocates a fixed sampling budget, where each sample of an item yields a noisy observation of its quality, with the objective of minimizing the expected opportunity cost of the selected set. We propose an approach that allocates each batch of samples by minimizing a surrogate objective derived from an upper bound on the expected opportunity cost. The approach is general: samples can be allocated in batches, the selection size $k$ is arbitrary, and it applies to many sampling distributions. It is also simple and intuitive. We prove that the method's posterior expected opportunity cost decays at an exponential rate of at least half the best rate achievable by any sampling rule. In simulations across a wide range of settings, the method is competitive with, and often improves upon, the best existing algorithm in each setting.

Mon 19 OctMachine Learning
The gist
Sometimes you need to pick the best few items from a group when you can only check their quality a limited number of times, and each check has some noise or error. The authors introduce an easy-to-understand method that decides how to split these checks among items to find the top choices with fewer mistakes. Their method works for any number of selected items and different types of quality measurements. They also prove their method is close to the best possible and show through tests that it often does better than existing approaches.
Open → 2610.26407v1

Adaptive sampling method cuts experiments to confirm good designs faster

Halve-Then-Greedy: Adaptive Sampling for Posterior Success with Near-Optimal Instance Dependence

Abstract: We study fixed-budget adaptive experimentation aimed at maximizing the probability of obtaining sufficient posterior evidence that a recommended design's mean performance exceeds a target. Unlike best-arm identification, this objective does not require finding the best design. For Gaussian bandits with a Gaussian prior, we propose \emph{Halve-Then-Greedy} (HTG), which combines a large initial balanced allocation, successive halving, and posterior-greedy sampling. We prove instance-dependent, finite-budget exponential bounds on posterior-failure probability that capture both separation from unfavorable arms and the presence of multiple favorable arms. On one-positive-arm families, we establish an asymptotic sample-complexity lower bound that HTG matches up to a logarithmic factor. We further analyze uniform sampling and sequential halving, revealing regimes where they require more samples than HTG by linear and logarithmic factors, respectively. Simulations illustrate HTG's sample-efficiency advantages and their dependence on instance structure.

Wed 14 OctMachine Learning
The gist
Sometimes, when trying to find a good choice among many options, it's enough to confirm one is good rather than find the absolute best. The authors study how to allocate limited tests to boost the chance of proving a chosen option meets a target. They introduce a smart way called Halve-Then-Greedy that starts broadly, narrows focus step by step, then samples where it counts most. Their method requires fewer tests than some existing approaches and comes with mathematical guarantees about its effectiveness.
Open → 2610.20997v1

Online treatment selection controls errors while learning from limited feedback

Online Individualized Treatment Selection under Partial Feedback: False Discovery Control and Minimax Optimality

Abstract: We study online individualized treatment selection when only the outcome under the assigned treatment is observed. Our objective is to preferentially treat individuals whose potential outcomes under treatment exceed a covariate-specific control reference, while limiting false discoveries among treated individuals. We propose Partial-Feedback Adaptive Individualized Treatment Selection (PAITS), which initializes learning from historical data and combines sequential estimation of local null risks, randomized exploration, and feedback-driven updates of the treatment selection threshold. Under regularity conditions, we establish asymptotic false discovery rate control and sublinear gap-weighted assignment regret relative to a population oracle. We further establish a minimax lower bound on assignment regret under prescribed uniform nuisance-estimation requirements and give conditions under which PAITS attains the corresponding constrained minimax rate up to logarithmic factors. These results quantify the tradeoff between accurate treatment assignment and nuisance learning under partial feedback. Simulation studies and a real-data analysis using the Jester recommendation dataset illustrate the finite-sample performance of PAITS.

Mon 12 OctMachine Learning
The gist
When deciding who should get a treatment based on their personal characteristics, we often only see the result for those who actually get the treatment, but not what would have happened otherwise. This paper studies a method called PAITS that learns over time to pick treatments for individuals while controlling mistakes where the treatment is not actually better. The method starts with existing data, tries some options randomly, and updates how it chooses treatments based on new results. The authors prove that PAITS balances learning well and making few errors, and show it works using simulations and one real dataset.
Open → 2610.16464v1

Estimation method gauges value of personalized policies in adaptive trials

Inference on the Value of Policy Improvement in Adaptive Experiments

Abstract: In this paper, we develop an asymptotically normal estimator for the value of policy improvement (VPI) under adaptively collected data. This value, which is defined as the gap between the value of the best treatment policy in a class and that of a baseline policy, gauges whether or not it is worth deploying a personalized treatment policy. Building an estimator for the VPI is naturally more difficult than estimating the value of a fixed policy. This is because, in addition to contending with the correlated data and decaying propensities associated with adaptive experiments, any estimator must also handle the non-regularity, or lack of pathwise differentiability, of the underlying estimand when the optimal policy is non-unique. Our proposed estimator works by sequentially fitting a weighted empirical welfare maximizing policy on historical data and then evaluating it using inverse propensity weighting on future samples. We neither assume that the optimal policy is unique nor make a margin assumption on the underlying heterogeneous treatment effects. A key difficulty in analyzing our estimator is showing that the regret incurred through policy learning is asymptotically smaller than the standard error of the estimator. We accomplish this by showing faster regret rates for policy learning under decaying propensities, which may be of independent interest. We not only evaluate the coverage of our estimator on a mix of synthetic and semi-synthetic data, but also employ it to estimate the VPI in an adaptive experiment evaluating the efficacy of job assistance programs in Jordan due to Caria et al. Although our estimates are generally more conservative than their hierarchical Bayesian analysis, which assumes that the optimal policy is unique, we still corroborate the benefits of personalization amongst Syrian refugees.

Sun 11 OctMachine Learning
The gist
This paper tackles the challenge of measuring how much better personalized treatments are compared to a standard approach in experiments where treatments adapt over time. The authors propose a new statistical method that accurately estimates this improvement without assuming the best treatment policy is unique or simple. They tested their method on various data sets and a real-world job assistance study in Jordan, confirming that personalized policies provide benefits. Their approach is more cautious but reliable than some existing methods.
Open → 2610.15789v1

Cross species neural patterns predict human drug effects in neurological disorders

Cross-species representation learning aligns mouse and human neural dynamics and tracks clinical drug efficacy

Abstract: Preclinical models poorly predict human drug efficacy, particularly in neurological disorders. Neural activity offers a uniquely rich source of translational information because it captures high-dimensional variation in nervous-system function that can be measured in both animals and humans. However, its high dimensionality makes it difficult to distinguish conserved disease-related features from variation arising from species, recording modality and experimental context. Here, we test whether shared neural dynamics can be identified directly from electrophysiology data by learning representations organized by biological state rather than species. We develop a dual-rule contrastive learning framework that aligns corresponding mouse and human states while preserving separation between distinct phenotypes. This framework recovered conserved sensory-response structure across species and, in epilepsy, resolved distinct relationships between three mouse models and heterogeneous human patient populations. When treated animals were projected into a frozen cross-species representation, drug-induced movement towards the human-aligned healthy state retrospectively tracked known clinical efficacy across ten model-drug combinations including a disease-specific detrimental effect. The framework also identified shared disease-associated neural dynamics between Fmr1-knockout mice and human 16p11.2 copy-number variant carriers despite differences in genetic aetiology and recording modality. Together, these findings show the potential of cross-species neural representation learning to map heterogeneous human disease onto experimentally tractable preclinical states and assess whether interventions restore human-relevant circuit function.

Thu 8 OctMachine Learning
The gist
It is hard to predict if a drug tested in animals will work in humans, especially for brain diseases. The authors created a method that finds similar patterns of brain activity between mice and humans. This helps spot shared disease features and shows when a drug moves animal brain activity closer to a healthy human state. Their method also found common brain activity changes in different genetic conditions affecting the brain. This approach may help test if new treatments will work in humans by comparing brain signals across species.
Open → 2610.11222v1

LinUCB learns efficiently by sampling like optimal experimental designs

Sampling Allocation of LinUCB: Optimal Design Limits in the Small-Gap Regime

Abstract: We study the sampling allocation of LinUCB in the small-gap regime, where the reward gaps are of order at most $n^{-1/2}$ over the decision horizon $n$. This scaling captures the hard instances underlying worst-case regret lower bounds, for which LinUCB is known to be near optimal up to logarithmic factors in $n$. Using a mean-field perspective, we characterize this allocation through the empirical sampling distribution, a macroscopic object that averages the effect of adaptive decisions over the horizon, and identify its limit as $n\to\infty$. We establish that in this regime, the empirical sampling distribution induced by LinUCB converges to the set of D-optimal designs. This central result reveals that, in the small-gap regime, LinUCB not only achieves near optimal minimax regret but also allocates samples in a way that is asymptotically efficient for learning the reward parameter, thereby connecting regret-driven online learning with information-efficient experimental design. Building on the optimal design limit, we obtain two useful consequences. First, we refine the asymptotic regret analysis of LinUCB in the small-gap regime by characterizing its leading-order constant in the limit. Second, we show that, despite LinUCB's adaptive sampling strategy, the regularized least-squares estimator satisfies a central-limit-type theorem in the small-gap regime, thereby enabling valid statistical inference for the reward parameter.

Mon 5 OctMachine Learning
The gist
Some algorithms decide which option to try next by learning from past choices. When the differences between options' rewards are very small, understanding how samples are allocated can help improve performance. The authors show that LinUCB, a popular decision algorithm, samples in a way that matches the best possible experiment designs when many choices are made. This means LinUCB not only minimizes losses but also gathers information efficiently to learn the true rewards. Their work also helps in making precise guesses about reward values and understanding how reliable those guesses are.
Open → 2610.06213v1

Clinical trials human AI interactions mapped with new multidimensional framework

Defining and Categorising Human-AI Interactions in Clinical Trials: A Multidimensional Human-AI Classification Approach

Abstract: This paper examines human-AI interactions (HAIIs) in clinical trials and presents a multidimensional categorisation framework that classifies interactions according to AI tasks, human-AI relationships, interaction configurations and interacting human groups. We define HAII, examine existing taxonomies and extend existing categorisation approaches through this novel multidimensional framework. We purposively sampled 15 clinical trials from a previously reported dataset. Each trial was independently categorised by two human reviewers and six large language model (LLM) classifiers. The proposed categorisation provides a structured method for the consistent identification, comparison and synthesis of human-AI interactions across clinical-trial records. The framework is intended to support more consistent comparison and synthesis of AI-related clinical trials and to make explicit the different forms of human involvement associated with AI interventions. The results demonstrate the potential for LLM-assisted categorisation while indicating the continuing importance of human judgement where trial records are incomplete or ambiguous. The principal contribution is a proposed multidimensional framework that brings together AI tasks, human-AI relationships, interaction configurations and interacting human groups within a single approach designed for clinical-trial records. Its significance lies in its potential to support more systematic identification, comparison and synthesis of how humans and AI interact in clinical trials.

Tue 29 SeptArtificial Intelligence
The gist
Understanding how humans work with AI in clinical trials can be confusing and inconsistent. This paper presents a new system that organizes these human-AI interactions by looking at the tasks AI does, how humans and AI relate, how they communicate, and who is involved. The authors tested their system on 15 clinical trials using human reviewers and AI models, showing it helps compare and understand these interactions better. Their work also shows AI can assist in categorizing trials, but human judgment remains important when records are unclear.
Open → 2609.38559v1

Virtual reality designs overlap in cognitive therapy for dementia

Game-Based versus Training-Based Virtual Reality for Cognitive Rehabilitation in Mild Cognitive Impairment and Dementia: A Systematization of Design Paradigms, Outcomes, and Evaluation Rigor

Abstract: Therapeutic exercises can help slow cognitive decline in dementia, but conventional programs are often tedious and difficult to sustain. Virtual reality (VR) offers an alternative through game-based approaches designed to improve engagement and training-based approaches focused on structured functional recovery. Prior reviews report inconsistent findings about these paradigms, while it remains unclear whether the labels represent systems that are technologically distinct. Following PRISMA 2020 guidelines, we systematize 34 studies published between 2015 and 2026, including 16 controlled efficacy trials and 18 design and feasibility studies involving mild cognitive impairment, dementia and Alzheimer's disease. We develop a taxonomy covering design paradigm, immersion, feedback modality, difficulty adaptation and hardware. Our analysis shows substantial overlap between game-based and training-based systems, suggesting that these labels do not clearly distinguish their technological characteristics. We also introduce VR-RES, a seven-dimensional framework for evaluating study quality. Its application highlights recurring limitations in the evidence base, particularly limited follow-up, poor reproducibility and insufficient analysis of demographic factors such as age, sex and education. Based on these findings, we provide recommendations for future VR cognitive rehabilitation research, emphasizing specific design features, rigorous evaluation and improved reporting practices.

Tue 29 SeptHuman-Computer Interaction
The gist
Helping people with mild cognitive problems or dementia stay mentally sharp can be hard because usual exercises are boring. The authors looked at many studies using virtual reality (VR) games or training programs to help these patients. They found that the differences between game-like and training-like VR are not clear because they often share features. Also, many studies don’t test long-term effects well or consider important factors like age or education. The authors suggest better ways to design and evaluate VR tools for brain health.
Open → 2609.38514v1

NeuronSifter improves planning of CNS interventions by modeling microenvironment dynamics

NeuronSifter: Intervention Planning in CNS Microenvironments

Abstract: Prioritizing central nervous system (CNS) interventions requires predicting how a dose, route, and schedule act on a partially observed microenvironment, then choosing the measurement that would change the decision. Action-conditioned predictors reduce a regimen to an identity token or a scalar exposure, discarding where and when the target is engaged; handing a point estimate to a separate planner then discards the joint uncertainty that makes a measurement worth running. We therefore treat decision quality as a property of the intervention interface, not of controller placement. NeuronSifter compiles regimens into state-conditional target-occupancy fields with support masks, propagates them through microenvironment dynamics with an occupancy-conditioned diffusion operator, and selects measurements by their expected reduction in intervention loss, assimilating typed outcomes into the same posterior. In a declared synthetic Alzheimer's disease (AD) evaluation over 64 paired scenario blocks, occupancy conditioning lowers trajectory continuous ranked probability score from 0.165 to 0.110 and raises intervention ordering accuracy from 0.760 to 0.880, and every paired benchmark contrast remains separated after Holm correction. Decision-directed acquisition attains terminal risk 0.160 against 0.166 for a matched numerical Bayesian experimental design planner, and reaches the target risk at 0.796 $[0.732,0.873]$ of an earlier design control's cost, while the corresponding ratio against the matched planner, 0.963 $[0.907,1.025]$, is not separated from equality; point-state and dependence-ablated interfaces instead raise risk to 0.220 and 0.199, and a full-posterior external controller ties exactly. Published AD trials supply a separate retrospective endpoint bridge.

Mon 28 SeptMachine Learning
The gist
Treating brain treatments as simple numbers hides important details about where and when drugs work in brain tissue. The authors created NeuronSifter, a method that models how treatments affect specific brain microenvironments over time and uses this to better plan interventions. By predicting how a treatment impacts brain cells and deciding the best measurements to improve decisions, NeuronSifter more accurately ranks treatment options. Tests in a simulated Alzheimer's disease setting show it improves prediction accuracy and lowers treatment risk compared to earlier methods.
Open → 2609.35445v1

Minimax policy improves decision making in bernoulli bandit problems

Multi-Armed Bernoulli Bandits via Minimax Single-Arm Stopping

Abstract: We develop an index policy for finite-horizon Bernoulli multi-armed bandits from minimax solutions to single-arm bandit (SAB) problems. Each SAB problem involves choosing between an unknown Bernoulli arm and a known reward. We show that minimizing worst-case regret of SAB problems over all non-anticipative policies admits an exact semi-infinite linear programming formulation. The resulting stopping policies offer a natural way to compare arms: the higher the known reward against which a policy continues sampling, the more promising the unknown arm. We turn this intuition into indices based on cumulative continuation probabilities, with a monotone adjustment and a reward-shortfall cap. By relating index errors to the regret of single-arm stopping policies, we establish a distribution-free regret bound of $4.45\sqrt{KT}+10.75K$ for $K$ arms and horizon $T$. This bound matches the minimax-optimal regret order established in the literature. The guarantee extends to rewards supported on $[0,1]$ through Bernoulli randomization. We also provide a finite-grid implementation with quantified approximation loss. In numerical experiments, the SAB-based index policy achieves lower worst-case regret than every tested benchmark policy across all evaluated numbers of arms and horizons, while closely matching the grid-based MAB minimax policy in the two-arm setting.

Sat 19 SeptMachine Learning
The gist
Choosing the best option among several uncertain choices is a common problem in fields like online advertising or clinical trials. The authors develop a strategy that focuses on single choices and compares them to known rewards to decide when to stop exploring. Their method guarantees good performance across all possible situations and works well when multiple choices are involved. Tests show this strategy outperforms existing methods, especially when dealing with several options over limited trials.
Open → 2609.22690v1

AI generated data can change causal experiment questions and results

When AI Generates Covariates: Causal Typing and Estimand Drift in Sequential Experiments

Abstract: AI-generated covariates from notes, conversations, images, and wearable streams can change the causal question when their roles are left unspecified. A generated feature may represent a treatment version, pre-action state, history, design variable, mediator, outcome proxy, observation process, or intercurrent event; these roles are not interchangeable. We formulate a causal type discipline for sequential experiments: a versioned representation map, a causal role classifier, a claim-status filter, and an estimand lock. The lock fixes a standardized proximal effect before generated covariates enter the analysis. Under audit correctness and standard identification assumptions, admissible role assignments preserve this estimand. We apply the established conditional-covariance characterization of compression bias to substitution of generated representations for design-relevant states. A standardized decomposition separates compression, conditional-law, and standardization drift. Further results cover mediator adjustment, post-action leakage, marker-intervention conflation, outcome-guided discovery, and state-measurement error. Cluster-level orthogonal estimators distinguish empirical and superpopulation targets under repeated sessions and missing outcomes. Simulations show that refinement helps when it retains design-relevant information, whereas design erasure, leakage, and same-data marker selection can produce bias or undercoverage. The framework places causal semantics and claim status before confirmatory inference with generated representations.

Tue 15 SeptArtificial Intelligence
The gist
When artificial intelligence creates data from things like notes or images, it can change what a study is actually measuring, especially in experiments done step-by-step. The authors explain that these AI-created features can play many different roles in a study, and confusing these roles can lead to mistakes in interpreting results. They propose a clear way to classify AI-generated data and keep track of what question the study answers. Their approach helps avoid errors that come from mixing up treatment effects with other types of data changes. They also tested their ideas with simulations showing their methods can improve analysis when the AI data keeps important information.
Open → 2609.17772v1

Evaluation methods guide healthcare large language model testing

A primer on evaluation methods for large language models in healthcare

Abstract: Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and not harm. This evaluation can be more challenging than traditional machine learning for many reasons, including probabilistic and open-ended outputs, and behavior that shifts with prompt design and accumulated context. This review covers four key areas of LLM evaluation: principles of study design, statistical methods, capability evaluation and clinical context evaluation. Capability evaluation considers different benchmarks, including multiple-choice, agentic and multi-turn benchmarks, alongside operational metrics like token usage. Clinical context evaluation addresses establishing accuracy of free text outputs, such as human review and LLM-as-a-judge, and clinical trial approaches. Across sections, we describe underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question. Together, this article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs.

Sun 13 SeptComputation and LanguageArtificial Intelligence
The gist
Large language models are increasingly used in medicine, but making sure they work well is tricky because they give complex and changing answers. The authors review important ways to test these models, including study design, statistical methods, and checking their abilities through quizzes and simulated conversations. They also discuss how to judge the clinical accuracy of the models’ free text answers, such as having humans or the models themselves evaluate outputs and running clinical trials. Their goal is to help people design better tests to make sure these AI tools help rather than harm patients.
Open → 2609.14819v1

Medical studies fall behind fast changing clinical ai models

The widening evaluation gap in medical large language model research 2023 to 2026

Abstract: Large language models are superseded every few quarters; clinical evidence takes years. We asked whether medical research is keeping pace with the systems it evaluates. PubMed returned 11,628 records for January 2023 to June 2026 across fourteen clinical domains, growing 45-fold; 2.5% used a randomised, controlled or prospective design. Evaluation lag, from a study's newest named model release to its own publication, widened from 1.33 to 6.08 quarters. Because discontinued models age mechanically, we benchmarked this against a counterfactual holding model composition fixed: migration to newer systems offset only 56% of the drift (95% CI 50-65). Randomised trials evaluated models a median 4.6 quarters older than other designs (P = 3 x 10^-19), yet among studies naming a model still under development no design differed from any other; 62% of randomised trials evaluated a discontinued family. Rigour and currency are in tension, and that tension reflects model selection rather than research timelines.

Thu 10 SeptComputation and Language
The gist
Medical AI models improve very quickly, but studies testing them take much longer to complete. The authors found that evaluations of these models are getting further and further behind the latest versions. Trials that use strict methods tend to test older models than other study types. This means there’s a trade-off between testing the newest AI and using rigorous study designs. The gap comes mostly from which AI models researchers choose to evaluate, not how long studies take.
Open → 2609.11770v1