Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization
2026-07-06 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how delays in getting information can make it hard for machines to learn the best actions in real life. They found that delays cause differences between what the machine thinks is happening and what is actually happening, which can mess up decision making. To fix this, they created a method called DUPO that uses a special model to understand and adjust for these delays. Their tests showed DUPO works better than older methods, even when delays are long and random.
reinforcement learningdelayed feedbackMarkov Decision Process (MDP)diffusion modelpolicy optimizationstochastic delaysstate estimationrobotic controluncertainty weighting
Authors
Junqi Tu, Zejiao Liu, Fangfei Li, Yang Tang
Abstract
Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.