Papers for

film editors

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Audio visual editing method improves lip sync in human videos

Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos

Abstract: Human-centric audio-visual editing requires precise alignment of lip movements, gestures, and expressions with speech. Existing methods process inputs through independent video and audio pipelines, preventing cross-modal interaction between visual editing and audio dubbing. Many of these pipelines further rely on mask-based inpainting. As a result, they suffer from poor lip-audio synchronization, cumulative pipeline errors, and an inherent mask-size dilemma. We present AVE, the first unified end-to-end framework that jointly performs reference-guided person swapping, insertion, and deletion across both audio and video modalities without requiring spatial masks. To overcome data scarcity, we design a synthesis pipeline and introduce AVE-Dataset, the first audio-visual editing dataset to provide 100K pairs with appearance and voice references. By combining Synthetic-to-Real reconstruction with preliminary audio warm-up, our framework transcends synthetic pipeline limits and mitigates audio-video token imbalance. AVE outperforms modality-separate audio-visual editing pipelines in editing realism, temporal consistency, and audio-visual alignment.

Tue 20 OctComputer Vision and Pattern Recognition
The gist
Videos of people speaking need their lip movements to match the sounds exactly. Past methods treated video and audio separately, leading to mistakes like lips not syncing well with speech. The authors created a new all-in-one system that edits both sound and image together without cutting out parts of the image. They also made a large collection of paired videos and sound samples to train the system. This approach improves how natural and synchronized the edited videos look and sound.
Open → 2610.27947v1

Audio video consistency measured at event level improves generation quality

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Abstract: As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/

Thu 15 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
When computer programs create videos with matching sounds, it's hard to check if the sound and video really fit well together, especially for complex scenes with many events happening. The authors designed a new way to check consistency between sound and video by focusing on individual events in the scene rather than the whole clip. They trained a model called AVC-Reward that automatically judges how well audio matches video, and it matches human opinions better than previous methods. This tool can help improve the quality of audio-video generation by guiding the process to make better-matched scenes.
Open → 2610.22321v1

Salient movie scenes identified by language patterns and token predictability

An Interpretable Linguistically-Grounded Analysis of Scene Saliency in Movie Screenplays

Abstract: We investigate the factors affecting salience in movie screenplay by posing two research questions: 1. {\em Does Uniform Information Density (UID) predict salient scenes?} 2. {\em What other linguistic properties distinguish salient scenes from non-salient background?} We answer both questions through a controlled corpus study on the MENSA corpus, paired with a Linguistically-Grounded Salience Model (LGSM), a dual-stream classifier that fuses a frozen BERT encoder with explicit linguistic features through adaptive gating. The UID picture is layered in long-form discourse. Per-scene UID summaries carry no independent signal once length is controlled, but at the token level salient scenes match non-salient material in the bulk while specifically suppressing the rarest, hardest-to-predict tokens in the upper tail. UID therefore lives at the within-scene token level rather than as a per-scene aggregate. Salient scenes also show a moderate and sustained cast of central characters (following an inverted-U pattern), strong third-person usage, syntactically simpler language, and more abstract vocabulary about motivation and conflict. LGSM exploits this signature and improves recall on salient scenes over a strong semantic baseline while yielding better downstream summaries.

Sun 11 OctComputation and Language
The gist
This paper studies what makes certain scenes in movie screenplays stand out, or be "salient," based on how predictable and complex their language is. The authors found that instead of overall predictability of the whole scene's words, the rare and hard-to-predict words inside the scene matter most for salience. They also show salient scenes tend to have a moderate number of main characters, simpler sentence structures, more third-person narration, and abstract words about motivations. Using these insights, the authors built a model that detects important scenes better than previous methods and helps create better summaries.
Open → 2610.15544v1