Hierarchical language memory improves robot task handling with less resources
HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation
Robotics
Summary
Robots often struggle to remember important details across different steps of a task, which can cause mistakes. The authors created a new system called Hierarchical Explicit Language Memory (HeLM) that helps robots keep track of these details using words, similar to how people talk to themselves in their heads. This approach lets robots handle more complex tasks involving many steps, and it works faster and uses less memory than older methods. The authors tested HeLM on many real-world tasks and found it improved success rates significantly.
What this means in practice
- •For robotics engineers: Enhance robotic manipulators with better memory for multi-step and context-dependent tasks, improving success rates and efficiency.
- •For industrial automation teams: Develop automation systems that maintain task context over long operations with less memory and faster processing, reducing hardware costs.
Authors
Gyeonghun Kim, Donghoon Kim, Minsoo Choi, Suyun Lee
Abstract
Recent advances in Vision-Language-Action (VLA) models have enabled fine manipulation capabilities, yet performing tasks that require working memory remains a key challenge. Most existing VLA architectures operate under a Markovian assumption, mapping only the current observation to an action, which leads to failure in non-Markovian environments. Inspired by the human 'inner monologue', we propose Hierarchical Explicit Language Memory (HeLM), a hierarchical VLA framework that maintains task-relevant history in explicit language through factorized event detection and event-triggered memory updates. We evaluate HeLM across 10 non-Markovian real-world tasks with 56 variants, including five newly formulated inter-episode tasks that require persistent context across task boundaries. HeLM improves intra-episode success by 12.8 percentage points over the strongest baseline and achieves 78% subtask success on compositional generalization tasks, while using up to 4.5x less memory and running 2.5x faster than the keyframe baseline at long horizons.