Framework improves humanoid robot skill failure detection and recovery
FRAMES: Failure Recovery And Monitoring of Embodied Skills for Humanoid Loco-Manipulation
RoboticsArtificial Intelligence
Summary
Humanoid robots can follow instructions by using language models to pick skills, but mistakes in moving or grasping can ruin complex tasks. The authors created FRAMES, a system that watches how a robot performs skills and stops it if something goes wrong. This system uses cameras and robot data to spot failures, then helps the robot recover. They tested the part that watches for failure in a virtual environment and found it correctly caught most problems and recognized when the robot worked well.
What this means in practice
- •For robot developers: Detect and recover from failures in humanoid robots performing complex manipulation tasks using monitoring of skill execution.
- •For robot simulation engineers: Improve robot control software testing by integrating failure-aware monitoring and recovery frameworks in simulation environments.
Authors
Ajay Vikram Periasami, Xinyuan Luo, Haoyu Li, Xianyi Cheng
Abstract
Large language model (LLM) planners can decompose natural-language instructions and select reusable robot skills, but choosing the correct skill does not guarantee successful physical execution. This gap is especially important in humanoid loco-manipulation, where errors during approach, grasping, transport, or placement can invalidate the remainder of a long-horizon plan. We present FRAMES, a failure-aware supervisory framework for the Unitree G1 humanoid that operates above the CEER whole-body controller. A Planner Agent selects subtasks through parameterized mid-level skills, while a vision-language-model-based Monitor Agent evaluates each skill using temporal multi-view observations and structured robot and contact evidence. Detected failures stop the active skill and provide grounded feedback to a Recovery Agent. The framework further includes a Memory Module for reusing prior skill experience, and geometric grounding via depth and segmentation. We independently evaluate the monitoring module of the framework in MuJoCo using 100 trials comprising 50 failed and 50 successful executions across five tasks. The monitor detects 48 of 50 failures, correctly accepts 46 of 50 successful executions, and achieves 94.0% overall accuracy. These results provide initial evidence for the monitoring component, while end-to-end evaluation of the complete recovery loop remains ongoing.