Hidden agent trajectories discovered to reveal interactions in dynamic systems
Structural Inference under Hidden Agents
Machine Learning
Summary
Sometimes, systems with many interacting agents are only partially observed, making it hard to understand how the hidden agents behave and influence others. The authors studied this problem where both the hidden agents' movements and their interactions need to be recovered from visible data. They developed a method called SIHA that starts without assumptions and then refines estimates of hidden agents and their connections by feeding information back and forth. Their approach improves predictions about agent behavior and uncovers hidden relationships, even in realistic scenarios like motion capture with occluded body parts.
What this means in practice
- •For robotics developers: Improve robot group coordination when some robots cannot be fully observed by reconstructing hidden movements and interaction patterns.
- •For surveillance system engineers: Enhance tracking and understanding of individuals in crowded scenes despite sensor occlusions or communication loss.
Authors
Zhongben Gong, Xiaoqun Wu, Mingyang Zhou, Hui Huang
Abstract
Recovering latent interaction structures from multi-agent dynamics is important for understanding and predicting interacting systems. Trajectory-based structural inference has achieved promising performance, but conventional formulations assume that the trajectories of all modeled agents are available. In practice, agents may become unobserved at deployment because of limited sensing, occlusion, or communication failure. Existing studies have considered unseen-node estimation, structural inference under partial observations, and missing-value imputation, yet the joint recovery of hidden-agent trajectories and their interactions remains underexplored. We formulate this problem as structural inference under hidden agents. Its key difficulty is a circular dependency: recovering interactions involving a hidden agent requires an estimate of its trajectory, while trajectory reconstruction can itself benefit from structural information. To address this challenge, we propose Structural Inference under Hidden Agents (SIHA), which combines structure-agnostic initialization with structure-guided iterative refinement. SIHA reconstructs hidden trajectories from visible observations, infers interactions using Neural Relational Inference, and feeds the estimated structure back into hidden-state reconstruction through multi-strength structural attention and iterative state--structure updates. Experiments on three benchmark dynamical systems demonstrate consistent improvements in visible-to-visible structural inference, while also showing benefits in hidden-state reconstruction and future prediction. Motion-capture experiments with simulated whole-limb occlusion further demonstrate its effectiveness in realistic hidden-agent settings.