InteractAgent enables long-term human motion tasks with adaptive correction
InteractAgent: Agentic Human Motion Interaction with Memory-Augmented Reflection
RoboticsComputer Vision and Pattern Recognition
Summary
Planning how humans move and interact with scenes over time is hard because it needs understanding of space, actions, and fixing mistakes along the way. The authors created InteractAgent, a system that plans, acts, and learns from each step to improve its motions using a single large language model combined with a motion model. This system watches its own actions, decides whether to continue or redo steps, and refines its plan as it goes. Their tests showed it worked better and caused fewer errors than other methods that do not learn from feedback.
What this means in practice
- •For robotics developers: Develop robots that can plan and correct their movements in complex indoor spaces by observing their actions and refining plans over time.
- •For game developers: Create realistic character animations that adaptively correct motion errors and interact accurately with new scene layouts without retraining.
Authors
Tao Zhang, Senhe Zhang, Zeyu Zhang, Yujia Zhang, Dong Gong
Abstract
Long-horizon human-scene interaction requires coordinated spatial reasoning, physical feasibility, and the ability to recover from compounding execution errors. Text-driven motion models are largely scene-agnostic and lack closed-loop correction, while geometry-conditioned approaches depend on paired scene-motion training or explicit 3D preprocessing that limits generalization to novel layouts and open-vocabulary goals. We present InteractAgent, an agentic multimodal LLM system for long-horizon human-scene interaction that plans, executes, observes, and reflects in a closed loop, using a single MLLM as both planner and evaluator and a frozen MotionLLM as the executor. InteractAgent integrates Online Observation for step-wise recovery decisions (continue/redo/replan) and Memory-Augmented Reflection for multi-round plan refinement with adaptive stopping. On UniHSI, InteractAgent improves task success and reduces penetration compared with training-free baselines. Code: https://github.com/AIGeeksGroup/InteractAgent.