Papers for
film editors
Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.
Audio visual editing method improves lip sync in human videos
Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos
Abstract: Human-centric audio-visual editing requires precise alignment of lip movements, gestures, and expressions with speech. Existing methods process inputs through independent video and audio pipelines, preventing cross-modal interaction between visual editing and audio dubbing. Many of these pipelines further rely on mask-based inpainting. As a result, they suffer from poor lip-audio synchronization, cumulative pipeline errors, and an inherent mask-size dilemma. We present AVE, the first unified end-to-end framework that jointly performs reference-guided person swapping, insertion, and deletion across both audio and video modalities without requiring spatial masks. To overcome data scarcity, we design a synthesis pipeline and introduce AVE-Dataset, the first audio-visual editing dataset to provide 100K pairs with appearance and voice references. By combining Synthetic-to-Real reconstruction with preliminary audio warm-up, our framework transcends synthetic pipeline limits and mitigates audio-video token imbalance. AVE outperforms modality-separate audio-visual editing pipelines in editing realism, temporal consistency, and audio-visual alignment.
Audio video consistency measured at event level improves generation quality
AVC-Reward: Event-Centric Audio-Video Consistency Evaluation
Abstract: As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/
Salient movie scenes identified by language patterns and token predictability
An Interpretable Linguistically-Grounded Analysis of Scene Saliency in Movie Screenplays
Abstract: We investigate the factors affecting salience in movie screenplay by posing two research questions: 1. {\em Does Uniform Information Density (UID) predict salient scenes?} 2. {\em What other linguistic properties distinguish salient scenes from non-salient background?} We answer both questions through a controlled corpus study on the MENSA corpus, paired with a Linguistically-Grounded Salience Model (LGSM), a dual-stream classifier that fuses a frozen BERT encoder with explicit linguistic features through adaptive gating. The UID picture is layered in long-form discourse. Per-scene UID summaries carry no independent signal once length is controlled, but at the token level salient scenes match non-salient material in the bulk while specifically suppressing the rarest, hardest-to-predict tokens in the upper tail. UID therefore lives at the within-scene token level rather than as a per-scene aggregate. Salient scenes also show a moderate and sustained cast of central characters (following an inverted-U pattern), strong third-person usage, syntactically simpler language, and more abstract vocabulary about motivation and conflict. LGSM exploits this signature and improves recall on salient scenes over a strong semantic baseline while yielding better downstream summaries.