EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created a new test called EgoGapBench to measure how well computers can choose the right action from a first-person point of view when other people are around. They found that humans do this well, but current AI models often pick actions done by others instead of the right one for themselves. Training AI on usual first-person data didn’t help much, but training on their new test data improved results, though not to human levels. This shows that understanding first-person perspective in action choice is different and harder than just seeing from a first-person view.
egocentric perspectivefirst-person viewaction selectionmulti-agent scenesmachine learningbenchmarkfine-tuningMLLMsegocentric action selection (EAS)
Authors
Jihyeok Jung, Jeewu Lee, Sanghyeop Kim, Chanhee Han, Seong Joon Oh
Abstract
Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when first-person body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent's perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection.