AI summaryⓘ
The authors tackle a problem where a complex and slow simulator is used to model oil reservoirs, but it's hard to adjust the simulator because you can't easily get gradient information or run it many times. They created a system that treats the simulator like a game environment and uses a type of AI called Soft Actor-Critic to learn how to pick reservoir parameters that best match observed data. Instead of replacing the simulator, their method smartly decides which simulations to run, saving time and ensuring physical accuracy. Their results show good performance on a benchmark problem and reveal more detailed insights than usual methods. Overall, this work demonstrates that reinforcement learning can effectively calibrate expensive scientific simulators with limited runs.
CalibrationPetroleum reservoirHistory matchingSoft Actor-CriticReinforcement learningPhysics-constrained policy searchSimulator-in-the-loopPorosityPermeabilityCMG IMEX
Authors
Nam-Phong Huu Nguyen, Duy-Dong Nguyen, Tho Quan
Abstract
Many scientific calibration problems expose only an expensive executable simulator, making gradients unavailable and large-scale training-data generation impractical. We study petroleum reservoir history matching as an instance of this broader AI problem and formulate it as physics-constrained, simulator-in-the-loop policy search. Our method wraps the CMG IMEX full-physics simulator as a Gymnasium environment and uses Soft Actor-Critic (SAC) to learn a stochastic proposal distribution over continuous porosity, directional-permeability, and well-skin parameters. Each interaction generates and executes a reservoir case, aligns simulated and observed production responses, and returns a reward that combines multi-response mismatch with penalties for physically invalid properties. Off-policy replay reuses costly simulator feedback, while maximum-entropy learning preserves exploration. Unlike forward-surrogate approaches, the policy learns where to evaluate rather than learning to replace the simulator; every retained candidate is validated by IMEX. Under a 200-call budget on PUNQ-S3, the best valid candidate achieves category-macro NMSE $0.0285$, $R^2=0.9324$, and a bounded match score of $97.23\%$. Well-level analysis further exposes localized water-rate failures hidden by pooled metrics. These results establish a full-physics proof of concept for reinforcement-learning-based calibration of expensive scientific simulators.