Large language models help teach robots to walk better using formal rules
From LLM-Generated Specifications to Learned Quadruped Locomotion
RoboticsArtificial Intelligence
Summary
Teaching four-legged robots to walk well usually requires carefully crafted reward rules, which can be hard to design. The researchers explored if large language models like GPT-5.5 and Qwen 3.6 could create these reward rules in a special formal language from simple natural language goals. They used these generated rules to train robot walking behaviors and found that some performed as well as expert-designed rules, especially when specifying walking styles. This approach helps reduce the manual work needed to teach robots complex movements.
quadruped locomotionreinforcement learningreward functionsSignal Temporal Logic (STL)Parametric STL (PSTL)large language modelsProximal Policy Optimization (PPO)MuJoCo simulationgait patternsbehavior specification
Authors
Merve Atasever, Keyan Azbijari, Cagan Bakirci, Alfredo Reina Corona, Tolga Izdas, Richard Yang, Erdem Biyik, Jyotirmoy V. Deshmukh
Abstract
Quadruped robot locomotion policies are often trained using reinforcement learning, which in turn relies heavily on hand-crafted reward functions. Designing reward functions requires substantial manual engineering, and it is often unclear which local rewards will induce the desired global behavior. Shaped rewards from formal specifications in languages like Signal Temporal Logic (STL) can make rewards more interpretable, but writing STL specifications itself still requires domain expertise. We study whether large language models (LLMs) can fill this gap by generating Parametric Signal Temporal Logic (PSTL) specifications that are subsequently used for policy learning. Given a natural language locomotion objective and a constrained specification grammar, GPT-5.5 and Qwen 3.6 independently propose STL templates for command tracking, safety, and gait structure. We instantiate the parameters of the generated PSTL templates using expert trajectories and retain only specifications that are consistent with demonstrated expert behavior. The resulting specifications are then transformed into smooth, finite-history reward functions and used to train a quadruped locomotion policy with Proximal Policy Optimization (PPO) in MuJoCo XLA (MJX). We evaluate both \emph{gait-aware} and \emph{gait-agnostic} settings. The former specifies walking-trot, trot, and bound regimes, while the latter allows contact patterns to emerge from the task objective. We compare against hand-engineered rewards, Text2Reward-style LLM-generated reward code, and an expert-switching oracle. Gait-aware Qwen 3.6 specifications achieved 100\% survival and command success across all tested speeds (0.3--2.1 m/s) and matched the target gait at high speeds, whereas Text2Reward achieved 0\% for both metrics at $\geq 1.9$ m/s. Videos: https://stl-locomotion.github.io/