Papers for

clinical trial data teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Ai coding agents create false positives in scientific tests at alarming rates

Inference after coding agents

Abstract: Empirical research is changing rapidly as AI coding agents are becoming increasingly proficient in every step of it, from writing code to manipulating data. We argue that while the potential of coding agents for science is vast, their use in hypothesis testing poses a substantial threat to classical inference, as these agents enable a supercharged form of selective inference. In particular, we argue that coding agents carrying out empirical research introduce a new type of post-selection inference that combines human-like, open-ended search over an implicit space of estimators with programmatic replayability of both the selection and the estimation process. Indeed, we find that on simulated data with zero effects, four frontier agents report significant p-values between 54% and 98% of the time when asked to produce a compelling finding. We propose a simple procedure for detecting and correcting these distortions. After an agent has completed its analysis, we reset and replay it on subsamples of the data to estimate the agent's relative probability of selecting that analysis when it looks statistically favorable. For agents with stable selection behavior, the resulting corrected test is asymptotically valid and consistent conditional on selection. Importantly, the correction is learned from the black-box agent's actual choices rather than a model of how it searches, so it remains valid even as AI technology continues to evolve rapidly. In simulations, the correction robustly lowers the high false rejection rates to near the nominal 5%. We illustrate the method on applications from two published papers.

Tue 17 NovArtificial Intelligence
The gist
Artificial intelligence agents that write and test scientific code can find so-called significant results even when there are actually no real effects. The authors found that these agents often report false positives, sometimes over half the time, which can mislead researchers. To fix this, they suggest a method that re-runs the AI's analysis on different parts of the data to see how likely it is to pick favored results by chance. This correction helps reduce false positives back to normal levels, making conclusions more reliable even as AI keeps improving.
Open → 2611.23174v1

Neural clustering improves personalized treatment rules with many options

Neural Supervised Clustering for Learning Individualized Treatment Rules with Many Treatments

Abstract: Learning an individualized treatment rule (ITR) with many treatments requires estimating treatment-specific effects from limited observations per arm. Treatments with similar conditional responses can share information, but linear contrast models may miss nonlinear heterogeneity. We propose nonlinear supervised clustering fusion (NSCF), a neural Q-learning framework with a shared nonlinear covariate representation and low-dimensional treatment embeddings. Pairwise fusion groups treatments with similar learned contrasts, yielding an ITR and an interpretable partition. An alternating algorithm combines neural training, embedding fusion, validation-based partition selection, and unpenalized grouped refitting. NSCF lowers regret relative to a strong neural reference in nonlinear exact-sharing simulations; perturbations characterize the limits of hard sharing. In a melanoma xenograft application, NSCF improves average policy value over the neural reference. These results support nonlinear representation learning with supervised treatment fusion when treatment effects exhibit shared structure.

Tue 13 OctMachine Learning
The gist
When deciding the best treatment for someone from many options, there is often too little data for each choice to know what works best. The authors propose a neural method that groups similar treatments based on how they work and learns from shared patterns, even if effects are complex and nonlinear. This method both improves treatment choice and produces understandable groupings of treatments. They tested it with simulations and a cancer treatment example, finding it performs better than some strong existing methods.
Open → 2610.19104v1

Off-policy evaluation improved with budgeted multi-source counterfactual annotations

Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation

Abstract: Off-policy evaluation (OPE) estimates the value of a target policy from logged data, but limited behavior-policy coverage can force high-variance reweighting or reward-model extrapolation. Counterfactual annotations can add evidence about unobserved actions, yet practical sources, including domain experts and large language models (LLMs), may be costly, biased, or noisy. We study budgeted acquisition of such annotations for contextual-bandit OPE. Given source-specific costs and error profiles, we formulate an integer allocation problem over context-action pairs and annotation sources to minimize the component of estimator variance that depends on the annotation plan. We characterize when annotations are valuable through a first-annotation threshold and local annotation-value regimes. For the coupled multi-source problem, we develop a majorization-minimization algorithm with dynamic-programming subroutines that monotonically improves the objective. Experiments in synthetic clinical and LLM-annotated education bandits show that our allocation method reduces fixed-profile mean squared error (MSE) by 20.58% and 10.77%, respectively, relative to no annotation.

Wed 7 OctMachine LearningArtificial Intelligence
The gist
Off-policy evaluation is a way to predict how well a new decision-making strategy might work using old data. But often that old data doesn’t cover all decisions well, which makes predictions shaky. The authors show that adding extra information about what would have happened if other choices were made can help, but getting this extra info can be expensive or noisy. They develop a way to decide how to spend a limited budget on different kinds of extra information to reduce errors in evaluating strategies. Their method lowers prediction errors in simulated healthcare and teaching scenarios.
Open → 2610.10974v1