Anticipatory Reinforcement Learning for Trajectory Tracking
2026-07-03 • Machine Learning
Machine LearningRobotics
AI summaryⓘ
The authors studied how to improve deep reinforcement learning for controlling a small helicopter. They added info about future targets and speeds to help the system anticipate actions instead of just reacting. In simulations, this improved accuracy a lot, but when tested on real hardware, the best results came from a simpler approach with less detailed predictions. This shows that very detailed future info isn’t always needed when applying these methods to real machines.
deep reinforcement learningpredictive controlproximal policy optimizationsim-to-real gap1-DoF helicopteranticipatory controlstate augmentationtracking errormean absolute deviationcontrol systems
Authors
Georg Schäfer, Jakob Rehrl, Stefan Huber, Simon Hirlaender
Abstract
Deep reinforcement learning (DRL) in industrial control often suffers from lag and overshoot due to purely reactive control based on the current tracking error. To achieve anticipatory control without high computational overhead, we introduce a predictive formulation that augments the DRL state space with target velocities and future reference horizons. Evaluating eight configurations using proximal policy optimization (PPO) on a 1-degree-of-freedom (1-DoF) helicopter testbed, simulation results showed a 9-fold error reduction, lowering the mean absolute deviation from 2.73° to 0.31°. However, zero-shot transfer to physical hardware revealed a sim-to-real gap. Interestingly, a simpler configuration using a single, further look-ahead horizon matched the real-world top performance of the most complex model (1.11°). Overall, evaluating various combinations of prediction horizons and target velocities demonstrated that highly granular predictive data is not necessarily required for physical transfer.