Ember-bench tests long-term cause and effect in household tasks
EMBER-Bench: Benchmarking Cross-Event Causal Memory in Long-Horizon Embodied Tasks
Computer Vision and Pattern Recognition
Summary
Many robots and AI agents need to understand how past events affect what they should do next, especially in long tasks around the house. The authors created EMBER-Bench, a new test with videos and questions that check if AI can figure out why certain actions are needed based on what happened before. They found most AI models struggle to connect past causes with future actions, showing there is still work to do. Humans performed much better, highlighting the challenge of causal reasoning in long tasks.
What this means in practice
- •For robotics developers: Validate and improve robot decision-making by testing their ability to connect past events to current actions in long household tasks.
- •For home automation engineers: Develop smarter home assistants that understand how previous events influence current states and actions over time.
Authors
Aoyang Cai, Boning Zhao, Shaoxuan Xie, Dahui Gao, Huan Yang, Zhongyuan Wang, Zhiwei Yu, Guocai Yao
Abstract
Lifelong physical agents must reason over extended interactions where past events continue to shape the world long after they disappear from view. Beyond recalling what happened, agents must infer how history changes the current state and constrains future actions. Yet existing embodied and video-memory benchmarks largely focus on historical retrieval and summary, leaving such history-dependent causal reasoning underexplored. We introduce EMBER-Bench, an egocentric benchmark for cross-event causal reasoning in long-horizon embodied tasks, for which we newly created the task design, video recording, and data annotation. It contains 189 household tasks and 699 QA pairs, spanning task progress, failure recovery, external interventions, and compound long-horizon tasks with distant dependencies and prerequisites, with fine-grained event and causal-chain annotations. EMBER-Bench evaluates reasoning in both directions: next-action prediction selects the next action from history, and causal traceback, given that action, identifies the historical event that makes it necessary. Input ablations that add action logs or privileged cause-and-consequence annotations to the video indicate which kind of historical information models fail to use. Among the 16 evaluated models, the highest overall accuracy is 61.2%, compared with a mean of 98.3% across two human evaluators. At paired decision points, correct traceback is not associated with correct next-action prediction. Adding action logs yields a gain of 1.6 points, whereas cause-and-consequence annotations yield an additional gain of 13.0 points on top of that. These results suggest that extracting causal information from past events and converting it into constraints on current actions remains a key difficulty for long-horizon embodied agents. Project Page: https://zhaoalexgoat.github.io/EMBER-Bench/