Flexible language edits improve robot task understanding and behavior
Flexible Language-Guided Representation Editing for Adaptive Robot Behavior
Robotics
Summary
Robots learn how to do tasks by watching demonstrations but don’t always understand the important reasons behind the actions. The authors propose a method that lets people use natural language to teach robots which parts of a task truly matter, by adding, generalizing, or ignoring features in the robot’s understanding. This method helps robots behave better and adapt more easily in different situations. They tested it on robot navigation and manipulation tasks with real language feedback, showing improved performance.
What this means in practice
- •For robotics engineers: Refine robot task behavior using natural language corrections to adapt to new or changing task requirements.
- •For automation integrators: Improve transferability of robotic task representations across different environments by iteratively editing features with human language feedback.
Authors
Audrey Lee, Nathaniel Dennler, Ilia Sucholutsky, Andreea Bobu
Abstract
Trajectory demonstrations show robots how to behave but not why, leaving them to guess which features of the task actually matter. Incorrect guesses produce misaligned internal representations and lead to incorrect robot behaviors. Because task features are often semantically defined, we argue that natural language is better suited for specifying features rather than relying on demonstrations. Although existing representation editing methods have proposed using language to specify new features, only allowing additive edits produces feature sets that are often over-specified or too narrowly scoped, especially in sequential editing tasks. We propose the Flexible Feature Editing Pipeline (FLEX), which uses language models to iteratively refine a robot's internal representation of task features from natural language corrections via three complementary operations: Add missing features, Generalize existing specific features, and Mask irrelevant features. We evaluate FLEX on 2D navigation scenarios, 7DoF Franka tabletop manipulation tasks, and a real-world study using naturalistic human language corrections and real-robot demonstrations. Comparisons against single-operation and unscaffolded baselines show that all three operations are necessary in interactive scenarios, and that FLEX achieves consistently higher normalized reward compared to other feature editing approaches while maintaining compact, transferable representations. Project available at https://mit-clear-lab.github.io/FLEX/