Papers for

video software developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Low-light video dataset captures motion with off-the-shelf cameras

PiLLVD: A Real-World Low-Light Video Dataset for Dynamic Scenes Captured with Off-the-Shelf Cameras

Abstract: Low-light image and video enhancement has received increasing attention in recent years. However, most existing enhancement methods are developed and evaluated on still images, while progress in low-light video enhancement remains limited by the lack of real-world paired video datasets. Acquiring paired low-light and normal-light video sequences is challenging because it requires spatially and temporally aligned captures while avoiding motion blur, exposure mismatch, and correspondence errors in dynamic scenes. To address this gap, we introduce PiLLVD, the Pi-based Low-Light Video Dataset, a paired low-light video dataset captured using an accessible Raspberry Pi-based multi-camera system with off-the-shelf cameras. PiLLVD provides paired low-light and aligned normal-light reference sequences for dynamic indoor and outdoor scenes, including diverse human and object motion under varying illumination conditions. Unlike existing datasets that are often image-based, static, synthetic, or limited to simple motion patterns, PiLLVD focuses on motion-rich scenes where frame-to-frame stability and motion-related artefacts are important for evaluation. In addition to presenting the dataset and acquisition pipeline, we benchmark representative pre-trained low-light video enhancement methods on PiLLVD under a zero-shot setting. The results show that existing methods do not generalise uniformly across the proposed sequences, highlighting the challenges posed by the proposed dataset and its value for both benchmarking existing methods and supporting the development of future low-light video enhancement algorithms.

Thu 12 NovComputer Vision and Pattern Recognition
The gist
Improving videos taken in the dark is hard, especially when things are moving. The authors provide a new collection of paired videos showing the same scenes in both low light and normal light, filmed with affordable cameras. This helps test how well current video enhancement tools work on moving scenes in low light. Their tests show existing methods struggle on these real-world videos, pointing to the need for better tools. The dataset is useful for those trying to make videos clearer in dark, dynamic situations.
Open → 2611.16515v1

Code2Video Compass benchmark evaluates code based video creation

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Abstract: Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.

Tue 20 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Creating videos by writing code instead of making them directly allows for more precise control and easier editing. The authors made a big test called Code2Video-Compass to see how well different AI models can create videos from text, images, or other videos using code. Their test covers many types of video tasks and checks video quality and how well the videos match the instructions. They found that even the best models still have trouble making good, fully accurate videos, especially when the tasks are more complex. This benchmark helps improve AI that makes videos through coding.
Open → 2610.28416v1

Speculative forcing speeds up video generation by skipping steps

Speculative Forcing for Efficient Autoregressive Video Generation

Abstract: Few-step autoregressive video generators enable streaming generation, yet they run every token through the final denoising step, regardless of how much refinement it needs. We find this uniform computation is often unnecessary: many tokens already closely match their eventual full-step outputs in the initial steps, while others require refinement until the end. We introduce Speculative Forcing, a framework that treats intermediate clean estimates as drafts and predicts when individual tokens can be safely accepted early. Once accepted, a token is frozen as its final output and undergoes no further denoising, while its cached keys and values remain available as context for the remaining tokens. We explore two alternative verifier designs: a training-free one based on similarity to clean content and stability across steps, and a learned one that reads the frozen generator's intermediate features to estimate each token's remaining gap to its full-step output. Across short-video, long-video, and world generation, Speculative Forcing preserves the quality of full-step sampling while substantially reducing computation. On LongLive, it matches the four-step generator with ~1.7 denoising steps per token, running over 1.5x faster at 41.7 and 37.8 fps. On Lingbot-World 2.0, where camera motion continually changes the visible content, it maintains quality with ~2.5 denoising steps per token and runs 1.35x faster at 24.5 fps.

Thu 15 OctComputer Vision and Pattern Recognition
The gist
Generating videos one piece at a time can be slow because every part is processed fully even if it doesn’t need much change. The authors found that some parts are good enough early on and don’t require more work. They created a method called Speculative Forcing that predicts which parts are ready to stop refining, so these parts are frozen and don’t slow down the rest. This approach keeps video quality high but uses fewer processing steps, making generation faster.
Open → 2610.22347v1

Outdoor low-light video dataset advances enhancement and segmentation

OLiVES: An Outdoor Low-Light Video Benchmark for Enhancement and Segmentation

Abstract: Low-light video processing remains an underexplored problem in computer vision, primarily due to the scarcity of high-quality ground-truth data. Existing public datasets exhibit several limitations, including restricted motion diversity, limited scene coverage, weak degradations, reliance on auxiliary modalities such as event data, and misalignment between low-light inputs and ground truth. Furthermore, current datasets typically address either low-light enhancement or scene understanding, whereas these tasks should be considered jointly. We introduce \textbf{OLiVES}, a benchmark dataset for outdoor low-light videos with dynamic scenes, designed to support both low-light video enhancement (LLVE) and video object segmentation (VOS). The dataset addresses key content gaps by incorporating a diverse range of natural textures, animals, occlusions, dynamic activities, and close-up scenes. To the best of our knowledge, OLiVES is the first outdoor low-light video dataset captured using professional cinema-grade cameras. It provides over 25,000 aligned normal- and low-light frame pairs, along with over 46,000 VOS annotations. We benchmark state-of-the-art LLVE and VOS methods on OLiVES and show that training on it improves performance on both tasks. The OLiVES project page is available at: https://anonymous-OLiVES.github.io/.

Wed 14 OctComputer Vision and Pattern Recognition
The gist
Processing videos taken in very dark outdoor settings is hard because good examples to learn from are missing. The authors created a new collection of videos filmed with high-quality cameras showing real scenes in low light, along with matching normal-light videos and detailed object outlines. This dataset helps improve computer programs that make dark videos clearer and identify moving objects in them. They also tested leading methods on this dataset and found that training with it makes these tasks work better.
Open → 2610.20501v1

Frequency step cache speeds up autoregressive video diffusion with better quality

Frequency-Step Cache: Separating Synchronization and Repair for Efficient Autoregressive Video Diffusion

Abstract: Autoregressive (AR) video generation enables long-horizon synthesis but requires repeated forward passes through large diffusion transformers. Training-free caching reduces this cost by reusing intermediate computation, yet existing controllers often couple two decisions: when to synchronize the cache exactly and how much computation to repair between synchronizations. We introduce Frequency-Step Cache (FSC), which separates these roles. FSC schedules exact full-computation anchors on a dense-to-sparse trajectory, while a lightweight latent-space risk based on frequency-resolved drift and token age controls inter-anchor repair through whole-step replay, selective token recomputation, or full repair. On SkyReels-V2, whole-step reuse provides most of the acceleration, while selective token computation is most effective as fidelity repair. A controlled ablation shows that using one reactive signal for both synchronization and repair reduces fidelity at matched latency, with the same separation advantage observed on FramePack and Matrix-Game. Across multiple autoregressive video backbones, FSC improves speed-quality trade-offs. On SkyReels-V2, FSC-Speed reaches 2.36x speedup versus MotionCache's 2.275x while improving SSIM by 0.019 and PSNR by 0.5 dB. At MotionCache-matched speed, FSC-Quality improves SSIM by 0.031 and PSNR by 1.0 dB and lowers LPIPS by 0.024. Without retuning, FSC-Quality transfers to SkyReels-V2 14B at 2.12x speedup versus 1.79x for MotionCache.

Sun 11 OctComputer Vision and Pattern RecognitionMachine Learning
The gist
Autoregressive video generation involves predicting video frames one after another, which can be very slow and computationally expensive. The authors present Frequency-Step Cache (FSC), a new method that cleverly separates when to fully recompute frames and how to fix small errors in between. This separation allows the system to run faster while maintaining or improving video quality compared to previous caching methods. They tested FSC on multiple video datasets and found it improves speed and quality trade-offs consistently.
Open → 2610.15922v1

Behavior pack optimization improves video question answering accuracy significantly

Behavior Pack Optimization for Video MLLM Post-Training

Abstract: Video multimodal large language models (MLLMs) keep climbing video question answering benchmarks, yet shuffling the frames, masking the segment that supports the answer, or occluding the target object barely changes their predictions. The accuracy rests on appearance and language priors, not on the temporal evidence the question asks for. We trace this to the unit of post-training: rewards are computed on a single response to the original clip, so the model is never asked to behave consistently across views. We propose Behavior Pack Optimization (BPO), which replaces the single response with a behavior pack of outputs across counterfactual views chosen by question type, scored jointly. The pack reward asks for stability when the intervention is irrelevant, sensitivity when key evidence is removed, and abstention when no evidence remains. To keep this objective stable at small pack sizes, BPO uses an anchor-relative advantage: the response on the original view serves as a per-prompt reference instead of a group mean over mixed views. On TempCompass, MVBench, and NExT-QA, BPO improves the macro accuracy of Qwen2.5-VL-7B-Instruct by 4.7 pp, the temporal-hard subset by 7.8 pp, and abstention F1 by 20.0 pp over a budget-matched vanilla GRPO baseline from the same SFT checkpoint. The gains transfer to Video-MME, LongVideoBench, and to LLaVA-Video-7B; ablations confirm they follow the view sets, not the rollout count. We hope this pack-level perspective offers a useful starting point for the video MLLM and multimodal post-training community as the field moves toward evidence-grounded video reasoning.

Fri 2 OctComputer Vision and Pattern Recognition
The gist
Video language models often get video questions right by guessing from visuals and language hints, not by understanding the video over time. The authors found this happens because existing training rewards only focus on single answers, ignoring how answers should change with altered videos. They created a method called Behavior Pack Optimization that trains the model to give consistent answers across different versions of a video clip, improving understanding of time-related details. Their approach led to better accuracy on video question tests and improved the model's ability to say when it doesn't have enough evidence to answer.
Open → 2610.03141v1

Video models unified to handle diverse tasks without retraining

Video, Ergo Genero: Unifying Video Tasks via Spatiotemporal Analogy

Abstract: Adapting video models to new tasks typically requires dedicated data curation and fine-tuning. While visual analogy provides a training-free alternative by specifying tasks in-context, it remains restricted to the image domain. To explore whether analogy-based methods can unify diverse video tasks and generalize to out-of-distribution scenarios, we introduce ViGeo, a framework that extends visual in-context learning to the video domain via spatiotemporal canvas completion. Evaluated on a diverse task taxonomy with a strict train-test split, ViGeo generalizes to unseen video manipulations and zero-shot modalities (e.g., event cameras). Finally, we identify task internalization, where a query format associated with a pretrained task overrides the demonstration, and show that this shortcut can be removed with a small amount of task-unrelated data, highlighting the need to decorrelate prompt format from task identity.

Sun 27 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Video models usually need special training for each new task, which can be slow and costly. The authors created a method called ViGeo that lets a video model solve many different tasks by using examples shown during the process, without extra training. This approach works even on types of video data the method hasn't seen before, like from event cameras. They also found a way to fix a problem where the model guesses the task based on the example format, rather than the example content.
Open → 2609.33935v1

UnStep speeds up video generation with fewer steps and no retraining

UnStep: Training-Free Acceleration of Causal Video Diffusion with Fewer Steps Than Distillation

Abstract: Distilling bidirectional multi-step video diffusion transformers into few-step causal models has become a common approach for streaming video generation. While these few-step students are significantly faster than the teachers they are distilled from, they remain slow for real-time generation. In this work we present UnStep, a training-free wrapper that accelerates few-step causal video models at inference by running them with fewer diffusion transformer (DiT) steps than during distillation and limiting the temporal window retained in the attention KV cache. We propose two inference-only mechanisms to recover quality lost by step reduction and attention windowing: renoising the generated latent frames to a near-clean level and reusing the existing clean-cache pass to refine them, and applying truncated SVD to the DiT attention value and output projections. We also accelerate inference with a quality-preserving runtime stack for the DiT and VAE decoder, including more efficient attention calls and KV indexing, fused Triton RoPE with cached coefficients, and VAE decoding with optimized memory layout, precision, and convolution kernels. By reducing computation and optimizing the runtime stack, UnStep sets a new throughput regime for causal video diffusion, by running substantially faster than current methods, reaching 50 FPS on a single H100 without quality loss, and 77 FPS on GB200, all without retraining.

Sat 26 SeptComputer Vision and Pattern Recognition
The gist
Generating videos using AI often takes a lot of computing time, which can make real-time video creation slow. The authors propose UnStep, a method that speeds up video diffusion models by running fewer processing steps and using smart tricks during generation without needing to retrain the models. They improve the quality by cleaning up the video frames and applying efficient math techniques. This approach makes video generation up to 50 frames per second on powerful hardware, making it much faster while keeping good output quality.
Open → 2609.32518v1

Trace benchmarks streaming video models by timing and response quality

TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

Abstract: Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying when evidence becomes valid, how visual history is maintained, or how responses are triggered. As a result, similar scores may correspond to different workloads, failure modes, and operational behavior. We introduce TRACE (Temporal Audit and Condition-aware Evaluation), a condition-aware benchmark and evaluation framework that makes these factors explicit. TRACE combines temporally audited visual tasks with evidence timing and instruction-dependent trigger annotations, a unified causal Core--Adapter protocol that controls information availability while recording actual history processing and response events, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. On 1,240 records from 517 videos, we evaluate eight publicly available models or systems in eight configurations. We find that nearly identical QA accuracy can mask substantial differences in completion, answer validity, and generation workload, while proactive performance separates into response quality, response delay, false alarms (responses emitted while no target window is currently valid and a later one remains), and missed target windows. These results show that streaming-video performance should be interpreted as execution-conditioned system behavior rather than a single score. Our benchmark and code can be accessed at \href{https://github.com/om-ai-lab/trace-bench}{https://github.com/om-ai-lab/trace-bench}.

Fri 25 SeptComputation and LanguageComputer Vision and Pattern Recognition
The gist
Understanding video as it streams is different from just getting one score at the end. The authors created TRACE, a way to check not only if video questions are answered right but also when answers come, how the video history is tracked, and how the models decide to respond. They tested popular models and found that similar scores can hide big differences like delays or wrong timing. This means to judge video AI better, we need to look at how they behave over time, not just their final accuracy.
Open → 2609.30670v1

Audio visual segmentation runs faster and simpler with encoder only model

Less is More: Encoder-only Audio-Visual Segmentation

Abstract: Audio-Visual Semantic Segmentation (AVSS) aims to identify, segment, and classify sound-emitting objects in video frames. Previous Transformer-based AVSS approaches largely inherit design principles from image segmentation models. Recent studies show that these image segmentation models contain redundant components that contribute little to the segmentation performance. Following this insight, we propose Encoder-only Audio-Visual Segmentation (EASE). EASE runs at up to 365 FPS, 3x faster than prior State-of-the-Art (SotA) AVS models at comparable accuracy, and trains in under 11 GPU-hours. Furthermore, we achieve SotA AVSS performance across different backbones and input resolutions. Our results demonstrate that AVSS can be both simpler and faster, providing a scalable foundation for future research and real-time applications. Code, model weights, and samples are available at https://ease-avs.notion.site

Thu 24 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Audio-Visual Semantic Segmentation is a process that finds and highlights objects making sounds in videos. The authors found that existing models use extra parts that don’t actually help much. They created a simpler model called EASE that is three times faster and just as accurate. This model also trains quickly and works well on different video qualities and sizes.
Open → 2609.29121v1

Video models contain correct motion data even when output is wrong

A Chosen Future Can Still Be Rewritten: Causal Writability in Video Models

Abstract: When a video model generates physically incorrect motion, did it fail to learn the correct motion, or did it learn it but fail to use it? We show the latter: the correct motion remains available inside the model and can still be made to control the generated video. We train on videos where red masses oscillate slowly and blue masses oscillate quickly, then test a red mass with fast observed motion. Even when the model generates slow motion in this conflicting case, a low-dimensional edit predicted from simple physical variables restores the correct fast motion. We call this ability causal writability. At fixed strength, we find a sharp depth boundary: the same edit changes the video before the boundary but not after it. This closure marks commitment for that write. The motion signal nevertheless remains, and a stronger downstream write can restore physical motion, while excessive gain overshoots. Early causal writability predicts which errors training later corrects: those errors are writable at more network depths than errors that persist. We reproduce both causal writability and its sharp closure in a pretrained 1.3B video model, supporting generality across model scale and training regime.

Mon 14 SeptMachine LearningComputer Vision and Pattern Recognition
The gist
Sometimes video models produce motions that look wrong, and it's unclear if they never learned the right motion or just aren't using it properly. The authors show that the models actually do have the correct motion inside them and that simple edits can make the videos move correctly. They discovered a sharp point in the model where changes stop affecting the output, marking when the model commits to a motion. This finding helps explain which errors can be fixed by further training and applies broadly to large video models.
Open → 2609.15980v1

Video-MOPD improves video understanding with multi-teacher learning

Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding

Abstract: Video understanding demands a convergence of complementary capabilities across perception, temporal understanding, and complex reasoning, which are difficult to jointly optimize within a single model. We introduce Video-MOPD-8B, an open-weight model dedicated to video understanding tasks. To fundamentally enhance its capabilities, we conduct targeted reinforcement learning (RL) optimization across three core domains: video temporal grounding (VTG), general video comprehension, and video STEM reasoning. We then unify their complementary capabilities via Multi-Teacher On-Policy Distillation (MOPD), which consolidates expert knowledge by supervising student-generated trajectories with routed teacher feedback. We further introduce Reliability-Aware Informative Sampling (RAIS), which selects examples with consistently reliable teacher supervision and large teacher-student performance gaps. Together, these components enable Video-MOPD-8B to achieve coordinated and comprehensive performance gains across diverse video understanding tasks. Extensive experiments on comprehensive benchmarks covering general video understanding, temporal grounding, video reasoning, and video STEM tasks demonstrate that Video-MOPD-8B achieves state-of-the-art performance among existing models at a comparable scale. The trained model weights are available at https://huggingface.co/LandH/Video-MOPD-8B.

Tue 8 SeptComputer Vision and Pattern Recognition
The gist
Understanding videos well requires a model to do several tricky tasks at once, like noticing details, understanding timing, and solving problems. The authors created Video-MOPD-8B, a model trained to get better at these by learning from multiple expert models each focused on a different skill. They use a special training method that combines feedback from these expert teachers while picking the best learning examples. This approach helps the model perform well across many types of video tasks.
Open → 2609.09300v1