Diffdreamer couples learned models with simulators for quadrotor control
DiffDreamer: Coupling World Models with Differentiable Simulation for Agile Quadrotor Control
Robotics
Summary
Flying fast drones through moving gates is very hard because the drone must predict how the environment changes while controlling itself very precisely. The authors created DiffDreamer, a method that learns about the environment and drone behavior separately and uses both a learned model and a physics simulator during training. This method helps the drone learn faster and perform better than other approaches. They tested DiffDreamer on tasks like aggressive flight maneuvers and showed it works on real drones too.
What this means in practice
- •For autonomous drone developers: Train quadrotor control policies that predict environment and robot physically for faster learning and better performance in dynamic racing tasks.
- •For robotics simulation engineers: Integrate learned world models with differentiable simulators to improve control policy optimization for agile robots in complex environments.
Authors
Fanxing Li, Yuxiang Huang, Shengyang Wang, Fangyu Sun, Shuyu Wu, Tianbao Zhang, Dexin Zuo, Yan Yufei, Danping Zou, Wenxian Yu
Abstract
Tasks such as quadrotor dynamic-gate racing require agents to anticipate external motion while maintaining precise high-frequency control. World-model-based reinforcement learning provides a natural solution by learning latent dynamics for future imagination, yet it often entangles environmental evolution and the robot's own dynamics in a single learned rollout, where model bias can affect policy optimization efficiency. We propose DiffDreamer, a differentiable reinforcement learning framework that tightly couples a learned world model with differentiable simulation for actor optimization. In one optimization step, the learned world model propagates latent states that capture external dynamics and history-dependent features, while the differentiable simulator propagates the robot's explicit physical state. We further decompose the actor objective into a critic-driven pathway for non-differentiable task rewards and a simulator-driven pathway for differentiable motion rewards. Experiments on aggressive tracking and dynamic-gate racing show that DiffDreamer improves training convergence over the model-free, model-based, and differentiable-RL baselines, and final performance over these baselines and classical planning and control methods. Additional experiments assess reward decomposition and the odometry-conditioned relative-state representation through ablations. We also provide proof-of-concept real-world flight experiments for validation.