Large vision language models detect human-like event boundaries in videos

Can an LVLM Perceive Event Boundaries Like a Human?

Computer Vision and Pattern Recognition

Summary

Figuring out where one event ends and another begins in a video is something humans do naturally. The authors studied whether large vision-language models (LVLMs) can spot these event boundaries like people do. They found that while these models miss many boundaries when working directly with video frames, they do well when refining candidate boundaries suggested by an unsupervised method. This approach works even for very long videos from a first-person perspective. They also created a new benchmark dataset called EgoBoundary to test event detection on long videos, where existing methods struggle more.

What this means in practice

Authors

Hyolim Kang, Jungsoo Lee, Yeji Song, Whie Jung, Janghoon Cho, Juntae Lee

Abstract

Large Vision-Language Models (LVLMs) reason effectively about video content, but whether they perceive its event boundaries in a manner consistent with humans remains unclear. We first investigate this through Generic Event Boundary Detection (GEBD), which identifies event boundaries naturally perceived by humans in continuous video, without relying on predefined event categories. On Kinetics-GEBD dataset, LVLMs miss many boundaries when predicting directly from frames, but effectively refine candidates proposed by an unsupervised discovery method, achieving state-of-the-art unsupervised GEBD performance. We extend this discovery-refinement approach to hour-long egocentric video, where boundary discovery requires a broader semantic scale and whole-video LVLM refinement becomes impractical. We address these challenges through coarse-to-fine temporal discovery and localized LVLM accept/reject judgments. Human evaluation shows that the LVLM's agreement with group consensus falls within the range observed among human annotators, with disagreements concentrated on low-consensus cases. Using this pipeline, we construct EgoBoundary, a benchmark for event boundary detection in hour-scale egocentric video. Representative supervised and unsupervised methods degrade substantially on EgoBoundary, highlighting the challenge of long-horizon event boundary detection.