Proactive AI systems watch and act at the right time in physical spaces
Proactive Physical AI through Continuous Observation and Timely Intervention
Computer Vision and Pattern RecognitionArtificial IntelligenceRobotics
Summary
Many AI systems wait for clear instructions before acting, but in the real world, important tasks can appear unexpectedly and change quickly. The authors study how AI can watch the physical world continuously and decide on its own when and how to help. They built a system called ProAgent that uses a timing model to spot when action is needed and then takes the right physical action or offers help. They also created a way to learn from lots of video data to improve this timing. Tests showed ProAgent acts accurately and avoids unnecessary actions, even with real robots.
What this means in practice
- •For robotics engineers: Enable robots to monitor environments and intervene timely without explicit commands, improving autonomous assistance in dynamic settings.
- •For smart home developers: Develop devices that proactively detect user needs or problems and act appropriately before explicit requests occur.$Commercial implications: This paper supports products that autonomously improve smart home responsiveness and user comfort by initiating timely interventions.
Authors
Zhenhua Yang, Jiafeng Liang, Xingyu Zhou, Yi Jiang, Xiayu Cao, Zui Chen, Luqi Wang, Haiyang Sun, Yufei Xiang, Chang Li, Rongchuan Mu, Bowen Chi, Zihe Tong, Mingyu Li, Huijie Ma, Yiqiao Yang, Baoqi Ren, Zhiming Ma, Zhihao Zhu, Jiahao Ji, Zhuoyao Wang, Qianyu Wang, Xiangyu Liu, Wang Zixiang, Shunliang Zhang, Ming Liu, Xiaocheng Feng, Duyu Tang, Bing Qin
Abstract
Physical AI is commonly formulated around an explicitly specified task: given a goal or instruction, a system perceives, reasons, and acts. In many real-world settings, however, important tasks emerge from continuously changing world states before a person explicitly formulates them. We study this complementary problem as Proactive Physical AI, where an AI system continuously observes the physical world, identifies situations that warrant intervention, and initiates appropriate actions at the right time. We develop ProAgent, an end-to-end system built around two components: a Jev-style proactive timing model (dubbed StreamingJev) that determines whether and when intervention is needed, and a Response Model that generates responses for people or invokes available device capabilities for physical execution. To support learning at scale, we develop an automated data pipeline that converts videos into supervision for proactive timing. Experiments demonstrate strong proactive performance under both explicit objectives and standing roles, combining more accurate intervention timing with fewer false triggers than the source-backbone baseline. Real-robot evaluations further demonstrate ProAgent's ability to identify emerging needs and translate them into appropriate physical actions.