Adaptive action sequence transfer predicts user steps in new environments
Action Sequence Transfer via LLMs for Heterogeneous Environments
Robotics
Summary
People often perform tasks step-by-step in one setting, but moving those exact steps to a different place with different objects can be tricky. The authors created a system that looks at the original actions and understands the objects and space involved, then predicts what actions would make sense in a new, different setting. They use detailed maps of the environments and ways to understand activities at different levels to help the system adapt. Their tests show the system can suggest reasonable action plans even when the new place looks very different.
What this means in practice
- •For home automation developers: Create smart assistants that adapt user task sequences from one home layout to another automatically.
- •For video game designers: Generate believable character action plans across varied game environments with different objects and spaces.
Authors
Choongho Chung, DongHwan Shin, Sung-Hee Lee
Abstract
We present an action sequence transfer system that adaptively transfers user action sequences across different target spaces. Given an input action sequence from a source space and scene graph representations of both the source and target environments, our system predicts a corresponding action sequence in the target space by adapting to the spatial and object constraints of the new environment. To achieve this, we leverage multi-level representations of user activity to generalize actions at varying levels of abstraction. To demonstrate our system, we collect a new scene graph-based dataset derived from the Ego4D GoalStep dataset for evaluation. Results indicate that our system can generate valid action sequences even between spaces with drastically different object configurations.