Soft robotic control learns faster using reduced mathematical models

Accelerated Reinforcement Learning for Soft Robotics: Hours Instead of Years

RoboticsArtificial IntelligenceMachine Learning

Summary

Teaching soft robots to move correctly usually takes a very long time because they are complex and hard to simulate. The authors use a mathematical method called adiabatic spectral submanifolds (aSSMs) to simplify the robot’s behavior into a much smaller, easier model. By training reinforcement learning on this small model instead of the full robot, they greatly speed up learning—from years to just hours. Their approach produces better control than training on large, complex neural networks, all while requiring far less data and time.

What this means in practice

  • For soft robotics engineers: Rapidly train control policies on reduced models for pneumatic soft arms, cutting hardware data collection time from years to hours.
  • For industrial automation teams: Improve soft robotic arm precision in dynamic tasks with faster policy development using mathematically reduced models instead of large neural networks.

Authors

Roshan S. Kaundinya, Hehui Zheng, Robert K. Katzschmann, George Haller

Abstract

Reinforcement learning (RL) requires large amounts of interaction data or a reliable simulator, both hard to obtain for nonlinear soft robotic hardware. Recent work shows that a time-scale separation inherent to soft robot dynamics justifies their reduction to low-dimensional invariant manifolds, known as adiabatic spectral submanifolds (aSSMs). We demonstrate that an aSSM-reduced model, a simple polynomial ordinary differential equation learned purely from pre-collected data, can serve as an RL training environment. On a pneumatically actuated soft arm, we train a trajectory-tracking policy in 3 hours on a six-dimensional aSSM-reduced model and deploy it without fine-tuning, whereas reproducing this policy training extrapolates to 3 years of hardware interaction. Our policy tracks dynamic targets with 40% lower error than otherwise identical policies trained on a 4.8-million-parameter recurrent neural network (RNN) or a 1.7-million-parameter transformer, which take 13 and 100 hours to identify, against 5 minutes for our mathematically justified 6,924-parameter model.