Cogwam aligns robot task reasoning with actions using semantic state
CogWAM: Aligning Semantic Cognition with World Action Modeling via Event-Driven Interfaces
RoboticsComputer Vision and Pattern Recognition
Summary
Robots can struggle to keep their actions and understanding of tasks in sync as they try to complete goals. The authors developed CogWAM, a method that keeps track of what a robot has already done and what it should do next using a special memory called Semantic State. This helps the robot better predict what will happen next and choose the right actions without losing track of the overall task. Their tests show CogWAM works well in simulations and real-world scenarios.
What this means in practice
- •For robotic system developers: Enable robots to maintain task context over multiple actions for improved complex task execution in real environments.
- •For automation engineers: Create more reliable dual-arm manipulation systems by reducing redundant task state updates during operation.
Authors
Sen Wang, Liu Liu, Xinjiang Wang, Zequn Chen, Haoyi Jiang, Taojun Ding, Tingyang Xiao, Zhizhong Su, Jie Wang, Sanping Zhou
Abstract
Robot policies increasingly incorporate semantic reasoning and future-world prediction, yet combining these capabilities does not guarantee that local predictions and actions remain aligned with task progress. We introduce CogWAM, a cognition-guided world-action model that establishes an explicit semantic interface between task reasoning and world-action learning through a persistent Semantic State, which stores completed task events and the active subtask. CogWAM updates this state only when observations indicate semantic transitions, allowing task-level context to persist across multiple action chunks. To bridge semantic context with physical prediction and control, CogWAM employs progress-conditioned WORLD and ACTION queries that selectively extract task-relevant information for future-world prediction and action generation. During training, the Semantic State provides shared task-progress context for both branches, while inference removes the future-prediction branch and directly generates actions from observations and the maintained state. We further introduce semantic training strategies to improve transition learning and closed-loop conditioning. Without additional robot-action pretraining, CogWAM achieves 15.56 / 11.70 % Score/SR on RoboDojo and state-of-the-art performance on BiCoord, while real-world experiments demonstrate closed-loop dual-arm manipulation with 16.4 fewer Semantic State regenerations than step-wise updating.