Language guided control improves robotic arm tasks in simulation
Instruction-Conditioned Motion Control for Robotic Arms: Language-to-Code Decomposition and Reinforcement Learning
RoboticsArtificial Intelligence
Summary
Robotic arms need clear instructions to do tasks like moving cubes or shaping letters. The authors introduce a method that breaks down language instructions into code that controls the robot, making it easier to trace how instructions affect actions. They tested this approach using a simulation with a robotic arm and saw improved success rates in tasks by training an instruction policy. They also created a way to independently check if tasks were completed correctly, separate from the robot's own feedback.
What this means in practice
- •For robotics engineers: Develop robotic arms that can interpret and execute spoken or written instructions with better traceability and success verification.
- •For automation system integrators: Use the language-to-code framework to improve programming and evaluation of industrial robot tasks involving object manipulation in simulation settings.
Authors
Fan Huang, Dezhong Tong, Changyu Chen, Haewoon Kwak, Jisun An
Abstract
Instruction-Conditioned Motion Control (ICMC) separates parameter specification, instruction generation, and code synthesis for language-driven robot control, making execution outcomes traceable to instructions and programs. We compare one-shot generation with ICMC under o1 and o4-mini on cube pick-and-place and letter-reshaping tasks for a UR5e arm in MuJoCo. We then train an open-weight instruction policy by GRPO through a frozen code generator on cube placement onto two- and five-cube letter layouts, averaging the rewards of five programs per instruction. The logged positive-reward rate rose from 31.6% to 35.3% (253/800 to 282/800) between the first and last five training steps. A log audit identifies positive binary labels associated with geometric-failure messages: at least 19 in that run, 18 in the dense-reward run, and 5 in the run with one program per instruction. In one case, all five rewards were 1.0 while the log reports a 0.9515 m miss against a 0.05 m tolerance; the verdict came from the program's own output stream. We distinguish logged reward rates from independently verified task success and report descriptive results for each configuration's single training trajectory. We introduce an independent evaluation protocol that checks task completion against fixed specifications, separating robot control from success assessment to support reproducible evaluation.