Model-based reinforcement learning improves flow controller design efficiency

Audited Gradients, Anchored Policies: Model-Based Reinforcement Learning for Efficient Flow Control

Machine Learning

Summary

Designing feedback controllers for flow control is costly because each option needs testing with detailed simulations. The authors introduce a way to check if a cheaper, approximate model’s information about how changes affect outcomes is reliable. If the model passes this check, they trust it to help find a good control strategy, otherwise they rely on the accurate but expensive simulation. This method leads to effective controllers using far fewer expensive simulations than traditional methods. They show that having a good prediction doesn't always mean the model can guide improvement, and that checks must be specific to where the controller acts.

What this means in practice

  • For industrial automation engineers: Reduce the number of expensive flow simulations needed to develop effective feedback control systems using audited surrogate models.
  • For aerospace system developers: Design efficient flow controllers with fewer costly computational fluid dynamics runs by anchoring policies to audited surrogate gradients.

Authors

Ruige Kong, Stefania Fresca

Abstract

Feedback controllers for flows are expensive to design, since each candidate must be evaluated with a high-fidelity solver. Differentiable surrogates reduce that cost, but they also introduce a failure mode that prediction error cannot detect. Policy optimization uses the derivative of the objective with respect to the action, which we call the action gradient and which can be computed either through the surrogate or through the high-fidelity solver. Optimization through the surrogate uses only its action gradient, and a surrogate can follow the flow trajectory closely while misestimating this derivative. The surrogate action gradient is compared against a central finite-difference estimate from the high-fidelity solver on held-out states, under acceptance criteria fixed before the comparison, and we call that comparison an audit. A low-dimensional feedback law is then fitted to the surrogate action gradient when the surrogate passes the audit, and to the solver estimate when it does not. This law initializes a neural policy, and a quadratic penalty holds the policy near it during optimization. The proposed approach is validated on four test cases. The resulting controllers outperform or match a model-free baseline, while using 3 to 156 times fewer high-fidelity solver interactions across the four test cases. One surrogate predicts the objective at correlation 0.9996 but its action gradient at only 0.459, so prediction accuracy does not identify a usable surrogate. Another surrogate, whose action gradient agrees with the solver estimate at correlation 0.996, makes the objective 0.52% worse when the policy is optimized through it without the anchor penalty, so an accurate gradient alone does not provide a working controller. Finally, a surrogate that passes the audit at one fixed action fails it around the actions the deployed controller applies, so a pass is local.