SymAnt improves robot prediction of long action sequences from short views
SymAnt: Grounding Long-Horizon Action Anticipation with Adaptable Symbolic Priors
RoboticsComputer Vision and Pattern Recognition
Summary
Robots need to guess what people will do next by watching only a little bit of video. The authors created SymAnt, a method that mixes smart symbols and neural networks to help robots understand scenes better. SymAnt looks at objects and their relationships using graphs to build a context before guessing future actions, making predictions easier to adjust. It works well even when the robot sees only a short clip and allows people to tweak its knowledge directly without retraining.
What this means in practice
- •For robotics engineers: Enable robots to predict extended human activities quickly from brief observations, improving interaction and assistance capabilities.
- •For human-computer interaction teams: Provide interpretable and editable models to anticipate user actions in real-world environments using symbolic reasoning over scenes.
Authors
Fnu Aryan, Yaqi Xie, Joseph Campbell, Woojun Kim, Kwonjoon Lee, Hossein Nourkhiz Mahjoub, Ehsan Moradi-Pari, Katia Sycara, Simon Stepputtis
Abstract
As robots and intelligent systems increasingly interact with humans, they must anticipate long sequences of future actions from only brief observations. While purely neural models typically require extensive observations to predict future actions and remain difficult to interpret and adjust, symbolic priors offer a structured alternative. To this end, we introduce SymAnt, a novel neuro-symbolic approach that unifies the expressiveness of neural networks with the structured reasoning of scene and knowledge graphs. Given a short video, SymAnt performs a joint-search over both graphs to construct a symbolic context, enabling unified reasoning over spatial relationships, object attributes, affordances, and associated actions. This symbolic context serves as a prior for an iterative refinement process that progressively generates and improves the sequence of future actions. By grounding predictions in symbolic context, SymAnt anticipates long-horizon actions from short observation windows in a parameter-efficient manner, while enabling users to directly edit the underlying knowledge to adjust predictions without further training. Through our experiments, we demonstrate the effectiveness of SymAnt, particularly when only short observation contexts are available, across five diverse datasets: Breakfast, 50 Salads, EPIC-KITCHENS, EGTEA Gaze+, and Assembly101.