Anticipating personal actions months ahead from first-person video
NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video
Artificial Intelligence
Summary
People often plan big but end up following habits and wish they could see their future actions before deciding. The authors collected a huge set of daily videos and data from one person over months to study if a computer can predict what they will do next. They created a test to see how well computers can guess future actions at different detail levels and times. This helps understand how far ahead a system can predict someone's behavior based on what they normally do.
What this means in practice
- •For personal assistant developers: Create assistants that preview likely user actions to help improve daily planning and decision making.$Commercial implications: Enables predictive personal AI products offering anticipatory suggestions based on long-term user behavior data.
- •For lifelogging app teams: Design tools that analyze extensive first-person data to forecast future activities with hierarchical detail.
Authors
Zhaoxu Meng, Yiming Sun, Mingyuan Gao, Jiachang Zhang, Zhuhan Dai, Yipeng Du, Zheng Lian, Jian-Qiao Zhu
Abstract
We often plan ambitiously yet act habitually and wonder, in retrospect, whether we would have planned differently had we known what we would actually do. Hindsight offers a valuable perspective on past decisions, although we often wish we could have simulated hindsight at the moment of choosing. If a system could generate plausible trajectories from one's personal history, such previews might help people formulate more realistic plans and make better informed decisions. We introduce NextMe-800, an approximately 800-hour first-person dataset from one volunteer over 126 days with 1 Hz images, gaze, and audio, captioned at five hierarchical abstraction levels from atomic actions to major activities. We formulate personalized action anticipation as open-vocabulary K-step sequence prediction and construct NextAct, a 1,500-point benchmark combining NextMe-800 with the multi-person EgoLife dataset. Using an embedding-based soft edit distance as the metric, we evaluate how well different models can anticipate personal behavior across abstraction levels and prediction horizons. NextMe-800 and NextAct provide a months-long resource and evaluation framework for studying how far ahead personal behavior can be anticipated from egocentric observation.