EgoPlay: Event-Triggered Video Editing for Egocentric Streams
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created EgoPlay, a video editing tool that changes egocentric (first-person) videos based on specific events described by the user, like "when X happens, do Y." Instead of using separate parts to detect events and edit videos, EgoPlay learns both tasks together in one model, making it more efficient and accurate. They trained it with a large dataset of clips tied to events and also made a way to measure how well it edits after an event without messing up the video before the event. Compared to previous methods, EgoPlay performs better in editing quality, visual consistency, and preserving backgrounds while using less computer memory.
egocentric videovideo-to-video editingevent-triggered editingdiffusion transformerEgo4D datasetend-to-end learningvideo diffusion modelevent detectionprompt-based editingtemporal video editing
Authors
Jinjie Mai, Gordon Guocheng Qian, Willi Menapace, Arpit Sahni, Chaoyang Wang, Ashkan Mirzaei, Runjia Li, Sergey Tulyakov, Bernard Ghanem, Peter Wonka, Rameen Abdal
Abstract
We introduce EgoPlay, an event-triggered video-to-video editor for egocentric streams, obtained by fine-tuning a pretrained V2V diffusion transformer on event-conditioned data built primarily from Ego4D. Given a monocular video and an event-triggered prompt of the form "when X happens, do Y," EgoPlay infers whether and when event X occurs, preserves pre-event frames, and applies edit Y only to the post-event continuation. Rather than cascading a separate event detector with an editor, EgoPlay learns event recognition, temporal restraint, and pixel-level editing jointly in a single end-to-end model, while also handling negative and multi-event prompts. To support this, we construct a large-scale dataset of 106K event-triggered clip-prompt pairs spanning positive triggers, fabricated-trigger negatives, and multi-event prompts. We then train a bidirectional video diffusion editor with event-triggered supervision and derive a causal variant for chunk-by-chunk streamable inference. We further introduce an event-aware evaluation protocol that separately measures post-trigger editing quality, pre-trigger preservation, and false-trigger robustness. On the Ego4D benchmark, EgoPlay substantially outperforms EgoEdit, the state-of-the-art instruction-based egocentric video editing baseline, with relative gains of 17.7%, 16.9%, and 16.4% in editing quality, visual quality, and background consistency. It also surpasses a VLM-guided detector-editor baseline by 15.7%, 14.5%, and 13.5% on the same metrics, while using less than half the GPU memory.