Multi-agent reinforcement learning cuts walking energy use with hip exoskeleton
Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction
RoboticsMachine Learning
Summary
Walking with assistive hip devices often costs extra energy to wear, reducing their usefulness. The authors developed a smart system that trains a hip exoskeleton and a detailed human model together in computer simulation to reduce the wearer's energy use. Their system learned a control policy that worked well without tuning for each person and reduced the real energy cost for eight test users. This approach allows testing different goals in simulation before trying on people, saving time and effort.
What this means in practice
- •For assistive device developers: Develop hip exoskeleton controllers that reduce metabolic cost for new users without individual tuning by training policies in simulation.$Commercial implications: The paper enables selling hip exoskeletons with ready-to-use energy-saving control software for diverse users, reducing setup time and costs.
- •For wearable robotics engineers: Evaluate and design exoskeleton assistance strategies in simulation with multi-agent learning before hardware deployment to predict metabolic benefits.
Authors
Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou
Abstract
Objective: This study aims to determine whether a hip-exoskeleton policy trained in simulation, by co-adapting with a musculoskeletal human model under a metabolic-energy reward, reduces the metabolic cost of walking in new users without per-user tuning, and how mechanical-power shaping affects the learned assistance. Methods: We developed MARL-Exo, a multi-agent reinforcement learning framework in which a full-body human model (304 muscle actuators; randomized anatomy, gait, and speed) and a bilateral hip-exoskeleton policy are trained concurrently in two stages. The exoskeleton observes only delayed hip kinematics and its past commands; its reward includes muscle metabolic power from a physiologically based energetics model, alone (Metabolic) or with mechanical-power shaping (Metabolic + Power). The Metabolic policy was deployed and tested without retraining in eight healthy adults walking at 1.25 m/s, using indirect calorimetry. Results: In simulation, the Metabolic policy reduced modeled muscle metabolic power by 13.8% relative to no-device walking, versus 8.8% for Metabolic + Power. On hardware, MARL-Exo reduced net metabolic rate by 18.3 $\pm$ 6.6% relative to Passive in all eight participants. Relative to No-Exo, the mean reduction was 7.4 $\pm$ 10.0%; six participants more than offset the 13.1% device-wearing penalty, walking up to 24.2% below their No-Exo cost. Conclusion: A metabolically informed policy co-adapted in simulation transferred to users without per-user tuning and consistently reduced the metabolic cost of walking with the device, while explicit power shaping reduced the simulated benefit. Significance: Unlike per-user human-in-the-loop optimization, this approach yields a single population-level controller and allows assistance objectives to be screened in simulation before human testing.