Combining symbolic reasoning and vision for better robot task handling
Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Robots can follow simple instructions well but struggle with complex tasks that need many steps and careful planning. The authors explore a method that mixes learned visual and language skills with clear rules about task order and conditions, helping the robot keep track of progress and make decisions. They also use hints from human demonstrations, like where people look, to improve robot actions. Tests in cleaning workspaces and handling surgical tools show this approach helps robots perform long, detailed procedures more reliably. This work shows how combining structured thinking with visual guidance can improve robot task execution.
vision-language-action (VLA) modelslong-horizon manipulationneuro-symbolic frameworktask graphsprocedural memoryconditional branchingobject groundinghuman demonstration guidancerobot teleoperationmultimodal reasoning
Authors
Vivek Chavan, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
Abstract
Vision-language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures requiring persistent task state, dependency-aware reasoning, conditional decisions, and reliable grounding. We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and multimodal procedural memory. Task graphs encode action dependencies, valid transitions, and branch conditions, while memory maintains the active step, completed actions, textual context, and task-relevant visual evidence. Together, these structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions. Human demonstrations provide additional spatial and temporal guidance through gaze or saliency cues. To isolate their effect on policy learning, our initial study bypasses cross-view gaze transfer and directly annotates pseudo-gaze in robot-view teleoperation videos. The resulting guidance is used during VLA fine-tuning and inference. We study two long-horizon manipulation domains, workspace clearing and surgical-instrument handling, which require ordered execution, visually grounded decisions, and conditional branching. We evaluate correct-object and destination selection, subtask completion, task progress, step-order consistency, complete-task success, and procedural or execution mistakes. This work positions structured symbolic reasoning and demonstration-derived visual guidance as complementary mechanisms for reliable long-horizon VLA manipulation.