Deep equilibrium flows improve robotic task diversity and precision
DEFlow: Deep Equilibrium Flow Maximum-Entropy Reinforcement Learning
Machine LearningRobotics
Summary
Choosing the best actions for robots in tricky tasks is hard when you want many good options, not just one. The authors create DEFlow, a new way for robots to pick actions by solving all steps together instead of one at a time. This helps robots perform better in tasks like docking and manipulation by carefully balancing variety and accuracy. They also show DEFlow can learn efficiently without expensive computations through special math tricks.
What this means in practice
- •For robotic control engineers: Improve robotic arm tasks like docking and object manipulation by enabling policies that better handle multiple good actions at once for precision and diversity.
- •For game ai developers: Create more realistic multi-action decision policies allowing game characters to choose between several strong moves efficiently.
Authors
Ruslan Ageev, Huaze Tang, Runqing Miao, Jiarong Li, Wenbo Ding
Abstract
Expressive flow policies can represent multiple high-value actions, but sampling them with few integration steps while retaining tractable action densities remains challenging. We introduce DEFlow, a deep equilibrium flow policy that replaces explicit Euler sampling with implicit Euler integration and solves all intermediate actions jointly as an equilibrium system. We combine implicit differentiation with a discrete change-of-variables formula for the final-action density, enabling off-policy training within soft actor-critic without backpropagating through solver iterations. DEFlow achieves an average relative improvement of approximately 10% in mean successful coverage over the strongest baseline across four Multi-Goal docking variants. It also achieves the highest mean time-averaged return on four of five manipulation tasks, while remaining broadly competitive across five MuJoCo tasks. These results support implicit flow policies for tasks requiring both diversity and precision, at the cost of additional nonlinear and linear solves. Primary