Pretrained offline reinforcement learning improves job shop scheduling results
PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problem
Machine LearningArtificial Intelligence
Summary
Scheduling jobs in factories to be efficient is very hard because many tasks and orders must be arranged well. The authors introduce a new method called PORL that first learns general scheduling rules by practicing in a simulated environment and then fine-tunes those rules using only past real factory data. This way, the method avoids relying fully on just simulation or just historical data, balancing exploration and real-world adaptation. PORL consistently performs better than existing offline learning methods, especially when the available historical data is limited or noisy.
What this means in practice
- •For industrial scheduling teams: Improve production schedules by adapting pretrained scheduling policies using historical factory data without costly online trials.
- •For supply chain optimization teams: Enhance scheduling decisions in supply chains by combining simulation-trained scheduling strategies with offline adjustments to real-world data.
Authors
Mateo Toro Diz, Jonathan Hoss, Noah Klarmann
Abstract
The Job Shop Scheduling Problem (JSSP) is a fundamental combinatorial optimization problem in industrial optimization. This work introduces Pretrained Offline Reinforcement Learning (PORL), a hybrid approach that combines simulation-based online pretraining with offline fine-tuning on production-specific data. Reinforcement learning through online interaction enables exploration of general scheduling strategies, but typically relies on simulation environments and may suffer from a simulation-to-reality gap. In contrast, offline RL avoids direct interaction with the environment by learning from historical data, but its performance is strongly influenced by dataset quality and coverage. PORL combines the strengths of both paradigms by first learning a general scheduling policy through online interaction and subsequently adapting it offline to a target distribution. A KL-divergence-based policy constraint is introduced to limit deviations from the pretrained policy during fine-tuning. The approach is evaluated on JSSP instances with distribution shift and datasets generated from heuristic, noisy-expert, and random behavioral policies. The results show that PORL consistently achieves lower optimality gaps than standalone offline RL and the considered general scheduling baselines. Furthermore, its advantage over standalone offline RL increases as dataset quality decreases, indicating reduced sensitivity to the quality and coverage of the available offline data. The results suggest that offline adaptation of pretrained policies is a promising approach for industrial scheduling environments where direct online exploration is impractical.