Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation
2026-08-17 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors studied how doctors decide to give fluids and vasopressors to patients with sepsis using data from ICU stays. They used a computer model that learns from past treatments to suggest better dosing decisions without testing directly on patients. By carefully checking their model’s reliability and comparing it to real doctor decisions, they found the model recommends slightly less fluid but is mostly similar to current practice. This suggests their approach could help improve sepsis treatment decisions in the future.
sepsisintravenous fluidsvasopressorsreinforcement learningoff-policy evaluationMarkov decision processpolicy iterationimportance samplingMIMIC-IV databaseclinical decision support
Authors
Marc Pérez-Roig, David Fernández-Narro, Carlos Sáez
Abstract
The dosing of intravenous fluids and vasopressors in sepsis is a sequential decision made under uncertainty and guided largely by clinical judgment, which makes it a natural target for reinforcement learning from historical care. Because a learned policy cannot be trialed on patients, its value must be estimated off-policy, and such estimates can be fragile and optimistic. This work advances the reliable evaluation of sepsis treatment policies by combining off-policy estimation, reliability diagnostics, and clinician-agreement analyses in a transparent validation framework. We modeled fluid and vasopressor dosing on a cohort of 36,872 septic ICU stays drawn from the MIMIC-IV critical-care database, as a discretized Markov decision process with 1,000 states and 25 actions, defined by a five-by-five grid of fluid and vasopressor levels and solved by policy iteration. The clinicians' behavior policy was estimated with a random forest, which mitigated the collapse of the Effective Sample Size (ESS 50.1 against 4.0 with smoothed counts) that otherwise destabilizes the importance-sampling estimate. The learned policy was evaluated with two estimators, weighted importance sampling (WIS) and fitted Q evaluation (FQE), with the ESS and clinician agreement as reliability checks. An empirical variable selection found that the composition of the state matters more than its size. Both estimators place the learned policy above the clinicians' return (WIS 50.8 and FQE 46.8 against 38.2, ESS 50.1), yet it departs only modestly from observed practice (total variation 0.18), favoring less intravenous fluid. These retrospective single-center off-policy results support the learned policy as a clinically plausible refinement of observed practice and motivate its further evaluation as a discordance-based clinical decision-support approach.