Summary
Controlling complex systems like traffic networks often benefits from combining AI that learns from experience with traditional model-based controllers that optimize actions based on known rules. The authors found that treating the traditional controller as part of the environment misses how the two controllers interact, which may limit learning. They propose a new method that explicitly considers the contributions and interactions of both controllers during training, allowing the AI and model-based parts to work better together. This new approach showed improved performance in some cases where the two controllers strongly influence each other, although the overall gains were modest.
What this means in practice
- •For traffic management teams: Better coordinate AI and traditional controllers to improve freeway traffic flow under complex interactions.
- •For robotic control engineers: Enhance autonomous robot control by jointly optimizing AI policies with model-based controllers when their controls strongly interact.
Authors
Giray Önür, Azita Dabiri, Bart De Schutter
Abstract
Integrated deep reinforcement learning (DRL) and model predictive control (MPC) methods are increasingly used to control autonomous systems by combining their complementary capabilities. DRL learns control policies through interaction with the environment. MPC uses a system model to optimize control inputs while accounting for constraints. In DRL-MPC frameworks with shared control authority, both the DRL agent and the MPC controller each determine part of the control inputs. However, common learning formulations treat MPC as part of the environment and therefore do not explicitly account for MPC's contribution to control or its interaction with the DRL agent. This paper proposes a novel composite-gradient learning (CGL) method that integrates the MPC controller into the learning process by representing the DRL and MPC control inputs as a joint action and accounting for their interaction when updating the DRL agent during training. CGL is evaluated on two multi-class freeway traffic networks with different strengths of interaction between the DRL and MPC control inputs and it is compared with alternative methods that treat MPC as part of the environment or that only partially incorporate MPC into learning. The results show that CGL offers limited benefit under weak interaction, but learns higher-performing control policies than the alternative methods in a subset of training runs under strong interaction, although the average control-performance gains remain modest.