Offline reinforcement learning improves with direct knowledge transfer
Direct Knowledge Transfer for Offline Q-Learning
Machine Learning
Summary
When data about how to make decisions is limited in one area, it helps to use information from related areas. The authors show a way to directly combine knowledge from other similar situations into the learning process. Their method, called Direct Transfer Fitted Q-Iteration (DFQI), links what is learned from other areas to improve decisions in the target area. They prove that this method works reliably and test it to show it helps when data is scarce.
What this means in practice
- •For marketing data teams: Improve campaign decision-making models in new markets by incorporating sales data from related regions.
- •For healthcare analytics teams: Enhance patient treatment policies in hospitals with limited records by transferring insights from other medical centers.
Authors
Zhanye Luo, Guangyan Gan, Lin William Cong
Abstract
Data-driven sequential decision making in marketing and healthcare can benefit from observations collected in related environments, particularly when data in the environment of interest are limited. Transferring such information requires accounting for differences in rewards and transition dynamics across environments. We study knowledge transfer for offline reinforcement learning through a joint estimation framework that directly integrates source information into target value-function learning. We propose Direct Transfer Fitted Q-Iteration (DFQI), which updates the target Q-function while simultaneously borrowing the source information through an L1 penalty that links the target coefficients to a weighted aggregate of source coefficients, with both determined jointly within each Bellman update. Using sieve approximation, we establish statistical guarantees for target estimation and policy learning in stationary discounted environments with common or heterogeneous transition kernels. Empirical studies further demonstrate the effectiveness of the proposed DFQI framework in transferring source information to improve policy learning with limited target data.