Calibrating black-box simulators using temporal-difference learning
Temporal-Difference Calibration of Black-Box Simulators
Machine Learning
Summary
Calibrating complex simulation models can be difficult when their inner workings are unknown or noisy. This paper presents a new method that adjusts simulator parameters by comparing the simulator’s short-term outcomes to real system data, without needing derivative information. The authors propose several algorithms that efficiently use noisy simulator outputs and prove they reliably find accurate parameter estimates. They also develop statistical tools to measure confidence in these estimates and add a step to further correct value differences without extra simulator runs.
What this means in practice
- •For simulation engineers: Tune parameters of complex simulators to better match observed system behavior using noisy and limited data.
- •For operations research teams: Improve decision policies by calibrating simulators to more accurately estimate value functions and outcomes.
Authors
Min Zeng, Xin Wen, Xi Chen, Yichen Zhang, Wen-Xin Zhou
Abstract
We develop a derivative-free temporal-difference (TD) framework for calibrating black-box simulators using one-step transitions from a target system and noisy simulator-generated continuation values. The calibration parameter is defined through Bellman orthogonality, recovering the physical parameter under correct simulator specification and yielding a near-optimal $L_2$ approximation of the policy value under misspecification. We propose three TD algorithms based on random-direction finite differences, distinguished by their query costs and reuse of noisy simulator evaluations. We establish almost-sure convergence, last-iterate mean-square error bounds, and central limit theorems for Polyak-Ruppert averages across four query-noise regimes. Our analysis reveals a specification-dependent bias phenomenon: for two algorithms, the finite-difference bias of the TD estimating equation vanishes at the calibration target under correct specification, even when the gradient surrogate remains biased. This permits a fixed perturbation radius and rate exponents approaching $1/2$ in the simulator rollout budget, whereas misspecification generally recovers the classical Kiefer-Wolfowitz rates. We also develop a few-replicate online bootstrap that yields Student-$t$ calibrated confidence intervals under negligible query noise, and identify a covariance mismatch that can invalidate conventional multiplier inference along Markov trajectories. Finally, a second-stage TD recursion corrects the remaining value discrepancy without additional simulator queries.