Papers for

video processing engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Video representation method speeds up training and playback significantly

Towards Seconds-level Video Representation: Quadratically Gated Feature Grid with Adaptive, Prediction-free Sampling

Abstract: Implicit neural video representations are notoriously slow to build, which does not cross the basic practical threshold for most industrial applications. Here, we present Q-GAS, a coordinate-queryable video representation that couples multi-resolution explicit 3D feature grids with a compact SwiGLU-style quadratic-gating (QG) decoder and an error-aware, prediction-free sampler. The explicit grids avoid hash-collision coefficient sharing at the cost of modestly higher storage yet greatly boost the expressive capacity; the QG decoder introduces low-rank local second-order interactions to improve the nonlinear feature response. During fitting, a global error map periodically refreshes a high-error pool, and each batch mixes pool samples with uniform samples, which is free from the auxiliary model to predict difficult samples, thereby further escalating the efficiency. On UVG Beauty/Bosphorus/Jockey, Q-GAS fits and quantizes under 50 s on one RTX PRO 6000, decodes at over 370 FPS with mixed precision and about 146 FPS in FP32. In our PyTorch implementation, Q-GAS trains and quantizes 5.5-7.2 times faster than the evaluated Instant-NGP baseline and 19-28 times faster than RNeRV. Under a 90 s budget, it improves PSNR/MS-SSIM/LPIPS over the evaluated NeRV-family pipelines at comparable BPP over the measured overlap. We have shared our code at https://github.com/MillenRosen/Q-GAS for readers' free use.

Fri 13 NovComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Building digital video representations that machines can understand and process usually takes a long time, which makes it tough for practical use in many industries. The authors created Q-GAS, a new way to represent video that uses multi-level feature grids and a special mathematical decoder to speed up both learning and playback. Their approach also uses a smart sampling method that focuses on errors without needing extra predictive models, making training more efficient. Tests show Q-GAS learns and compresses videos much faster than some recent methods while keeping good video quality and fast playback speeds.
Open → 2611.17786v1

New encoding improves color quality in sdr to hdr video conversion

Dark Matters: VAE-Aware HDR Encoding for SDR-to-HDR Video Reconstruction

Abstract: SDR-to-HDR reconstruction is an ill-posed problem in which clipped highlights, crushed shadows, and quantization artifacts must be recovered while preserving the original content. Latent diffusion models provide powerful generative priors for restoring missing detail, but their effectiveness is constrained by how HDR signals pass through the pretrained VAE. We find that common log-like encodings suppress low-luminance chromatic differences before generation, producing a dark-side collapse in the frozen-VAE latent. We introduce EquivFlow, a compact, invertible exposure-equivariant encoding that maps exposure changes to additive shifts while retaining a scale-independent chromatic component, without modifying the VAE. Encoding-level tests across pretrained image and video VAEs show that its exposure structure and deep-dark chroma transport persist across the tested hosts. Integrated into SDR-to-HDR video diffusion, EquivFlow delivers consistent chromatic-fidelity gains across the tested settings while retaining competitive reconstruction fidelity.

Tue 10 NovComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Converting standard dynamic range (SDR) videos to high dynamic range (HDR) is tricky because important details like bright highlights and deep shadows are often lost or distorted. The authors found that commonly used methods struggle because they lose color details in very dark areas during processing. They created a new mathematical encoding called EquivFlow that keeps color information intact even in those dark parts, without needing to change existing models. This improves the quality and color accuracy when turning SDR videos into HDR.
Open → 2611.13504v1

Training free real time video editing keeps source details intact

AR-Edit: Training-Free Autoregressive Video Editing without Inversion

Abstract: Editing a video stream in real time, without access to future frames, is essential for interactive applications. Existing training-free video editing methods, however, assume full offline access to the entire sequence, while streaming approaches either rely on additional training or are limited to specific editing techniques. We show that this limitation can be removed by analyzing the self-attention dynamics of DMD-distilled autoregressive (AR) video diffusion models. We uncover a key asymmetry: queries and keys remain stable and aligned with their source counterparts across denoising timesteps, while values diverge and encode the evolving content. Inherited from the non-distilled bidirectional teacher, this asymmetry is strongest at high noise, where few-step streaming models operate. The resulting mechanism is simple: a single forward pass suffices to capture source identity in the value features, which can be reused throughout generation. Starting from pure Gaussian noise and injecting only these source values reconstructs the input with fidelity matching or surpassing inversion-based methods, effectively eliminating inversion. Based on this insight, we introduce AR-Edit, a training-free, inversion-free method that caches source values once and selectively injects them during denoising. Across multiple AR backbones, AR-Edit enables real-time streaming editing with superior source preservation and tonal fidelity at minimal overhead.

Fri 6 NovComputer Vision and Pattern Recognition
The gist
Editing videos as they play, without seeing future frames, is hard but useful for things like live streaming. The authors discovered that a special kind of video model keeps some information stable, which lets them reuse parts of the video data for editing without needing extra work to invert or understand the original video. This means they can edit videos quickly and accurately while keeping the original content's look and feel. Their method, called AR-Edit, works in real time and doesn’t require training on each new video.
Open → 2611.09795v1

Tri-axis cosine transform compresses medical 3D image memory use

TACT-KV: Tri-Axis Cosine Transform for Compressing Volumetric KV Caches in Medical VLMs

Abstract: Vision-language models (VLMs) applied to 3D medical scans face a structural memory wall. A single CT or MRI volume produces tens of thousands of vision tokens, and the resulting key-value (KV) cache scales linearly with both batch and context. Existing KV compression methods operate on the cache's feature or token statistics, or apply 1D and 2D transforms, leaving the three-axis spatial structure in the cache unexploited. We propose TACT-KV (Tri-Axis Cosine Transform), a calibration-free KV compression method that exploits strong spatial correlations in visual KV states along all three anatomical axes of 3D scans. This correlation concentrates the KV cache energy under a frequency-domain transform. TACT-KV applies a separable 3D discrete cosine transform to the KV cache, allocates a per-layer bit budget across frequency bins via reverse water-filling (more bits to high-energy components), and quantizes each bin with a fixed scalar codebook. We derive a close-form relation: the distortion reduction from adaptive bit allocation is dominated by the spectral concentration of the transformed cache. Across six medical VLMs and two 3D CT benchmarks, TACT-KV achieves the lowest 1-bit prediction-flip rate on eleven of twelve model and dataset pairs in our main comparison. The same advantage reproduces on four general-purpose VLMs evaluated on video tasks, indicating that the benefit of exploiting three-axis structure generalizes beyond medical imaging. At equal high-bandwidth memory (HBM), TACT-KV scales the decoding batch size on a single GPU by up to $8\times$.

Thu 5 NovComputer Vision and Pattern RecognitionMachine Learning
The gist
Handling 3D medical images like CT or MRI scans uses a lot of computer memory because they produce many visual tokens. The authors found a way to compress this memory by looking at patterns along all three directions in the images, which other methods ignored. Their approach uses a special mathematical transform to keep the most important information while saving space, letting computers process more images faster. This method works well not only for medical images but also for video tasks, making it broadly useful.
Open → 2611.07593v1

Decoupled sparse attention speeds up video diffusion without training

DecoupAttn: Decoupled Sparse Attention for Training-Free Acceleration of Video Diffusion

Abstract: Sparse attention accelerates few-step video diffusion transformers by computing a few key blocks exactly and compensating for the rest. However, existing sparse attention still faces key challenges, including identification overhead from finding critical blocks at every denoising step, suboptimal budget allocation that sets sparsity budgets uniformly or by local metrics, and cross-step error accumulation as reused masks or clusters go stale. Observing that the errors left on skipped blocks cancel each other once every key is compensated, we propose DecoupAttn, a new paradigm of decoupled sparse attention: semantic clusters compensate every key, and exact attention corrects their error on a few blocks fixed once offline by position. To support this paradigm, we introduce Distortion Budget, a rate-distortion objective for budget allocation that places exact computation across timesteps, layers, and heads where it removes the most distortion that compensation leaves in the final latent. Built on this budget, we further propose Reassign Cache, in which each step reassigns the drifting keys to cached centroids and refreshes compensation while the budgeted blocks stay fixed. Moreover, DecoupAttn is training-free: a few calibration clips compile one plan offline, and inference runs standard block-sparse kernels with one fused compensation kernel. Our code will be released after submission.

Wed 4 NovComputer Vision and Pattern Recognition
The gist
Processing videos with diffusion models is slow because it requires computing many attention steps. The authors found a way to skip less important parts and correct the errors efficiently without retraining the model. They do this by splitting attention work between fixed important parts and groups that compensate for skipped parts over time. This approach is set up once using calibration videos and then speeds up video generation during use.
Open → 2611.05917v1

Monocular depth models face challenges in viewpoint consistency and metric anchoring

Depth Anyviewpoint? Evaluating Monocular Depth Over Viewpoint Sequences

Abstract: Monocular depth estimation aims to predict depth maps from single RGB images (forgoing the need for stereo camera pairs) and has seen impressive leaps in performance in recent work due to the introduction of affine-invariant losses and the use of foundation model features. These monocular depth models predict affine-invariant depth, which describes the relative positions of objects in the scene instead of absolute depth. While these models demonstrate strong performance on single image benchmarks and qualitatively consistent predictions in video, their effectiveness at preserving consistent spatial structure has been less thoroughly explored. Many potential real-world applications, such as handling video streams as input, rely on coherent three dimensional consistency in scenes. Further, anchoring relative depth predictions to a metric reference for a given scene is a key challenge for applying monocular depth models to real-world situations. We develop a novel evaluation protocol for monocular depth models using video sequences with moving camera positions to test the consistency of prediction across variation in viewpoint. We demonstrate this methodology over diverse, real-world video sequences where we are able to anchor depth model output to metric depth values using structure from motion. Our work demonstrates open challenges in the development of broadly applicable monocular depth foundation models, particularly with their ability to provide temporal consistency over long time frames. We also identify challenges in the straightforward adoption of structure-from-motion as a metric grounding tool for monocular depth estimation models.

Sun 1 NovComputer Vision and Pattern Recognition
The gist
Predicting how far away things are from a single photo is useful but tricky because the output often only shows relative distances, not exact ones. The authors studied how well these models keep consistent distance predictions when the camera moves around a scene in a video. They created a new way to test this and found that current models struggle to keep depth predictions steady over time and changing views. They also found that using techniques that reconstruct 3D scenes from video to set exact distance scales is not simple. This work highlights the challenges that remain for monocular depth models to be reliable in real-world, dynamic settings.
Open → 2611.02203v1

Video large language models get faster without retraining costs

Training-Free Inference Acceleration for Video Large Language Models: Methods, Trade-offs, and Evaluation

Abstract: Video Large Language Models (VLMs) have demonstrated strong capabilities in video understanding and reasoning, but processing long visual sequences incurs substantial computational, latency, and memory overhead. This survey examines training-free inference acceleration, focusing on reducing inference costs while preserving task-relevant spatiotemporal evidence. We organize existing methods by their intervention stages in the inference pipeline, covering input, visual encoding, projector, prefill, and decoding. Post-training quantization (PTQ) is discussed separately as a cross-stage extension with varying calibration requirements. We analyze how intervention position affects evidence preservation and computational savings, and why local efficiency improvements may not translate into end-to-end acceleration. We complement this analysis with quantitative evidence from prior studies and our own experiments under a shared evaluation setting to assess accuracy-latency trade-offs.

Mon 26 OctComputer Vision and Pattern Recognition
The gist
Processing long videos with large language models is very slow and uses a lot of computer memory. The authors review different ways to speed up these models during their use without needing extra training. They show how changing parts of the model’s steps affects both speed and accuracy, and why some improvements don't always make the whole process faster. They also compare past methods and run their own tests to find the best balance between speed and performance.
Open → 2610.36327v1

Training free allocation boosts video diffusion transformer attention quality

Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers

Abstract: Full 3D attention is costly in video Diffusion Transformers. Block-sparse attention reduces this cost by computing only selected attention blocks. However, existing methods either overlook the potential to optimize sparsity allocation or rely on heuristic rules or local metrics to guide it. In this work, we observe that sparsifying attention in different layers and at different denoising steps affects final-video fidelity to substantially different degrees. Yet the identities of the most sensitive layer-step pairs remain nearly unchanged across inputs. Building on these findings, we propose Allocating Layer-step Latency by Output Tests (ALLOT), a training-free sparsity allocation method. Treating each layer-step pair as a cell, ALLOT estimates each cell's contribution to final-video error through (i) cell probing and (ii) trajectory calibration, then jointly optimizes cell densities under a measured-latency budget. It preserves existing selection strategies and kernels, replacing only the allocation component. At equal or shorter wall-clock time, ALLOT outperforms every compared allocation strategy with XAttention, with an 11.8 dB PSNR gain on HunyuanVideo. Extensive experiments across four additional published pipelines on Wan2.1, Wan2.2, and HunyuanVideo show that ALLOT outperforms or ties the default in all nine settings.

Wed 21 OctComputer Vision and Pattern Recognition
The gist
Video diffusion transformers use a process called attention to analyze video data, but doing this fully is very slow. The authors found that paying attention (sparsity allocation) differently across parts and times in the model affects video quality a lot, and the important parts stay the same for different videos. They created ALLOT, a method that decides how to distribute this attention without needing extra training by testing each part's impact on video quality. This approach speeds up video processing while improving quality compared to other methods.
Open → 2610.29441v1

Balanced diverse video dataset enables improved video super-resolution

SCOPE-VSR: A Broad-Coverage Training Dataset for Video Super-Resolution

Abstract: Open-domain video super-resolution (VSR) requires temporally coherent details across widely varying scenes and subjects. However, existing training datasets are often semantically imbalanced and insufficiently diverse. We address this data bottleneck with a structured curation pipeline for open-domain VSR, SCOPE. In this pipeline, we combine task-oriented video selection with hierarchical semantic rebalancing. Applying this pipeline to a large-scale collection of data sources, we construct the SCOPE-VSR dataset for training and SCOPE-test for testing. SCOPE-VSR has approximately 27k videos and provides balanced and diverse coverage of various real-world scenarios and subjects. We further develop GraCo-VSR, a novel framework for one-step video restoration. GraCo-VSR adopts a reliability-aware design with explicit supervision to reduce degradation-induced distortions in motion and texture guidance, allowing the model to benefit more effectively from the diverse training content in SCOPE-VSR. Evaluations show that our model achieves strong reconstruction fidelity and perceptual quality. Our code is available at: https://github.com/gyliao2768/SCOPE-VSR.

Sat 17 OctComputer Vision and Pattern Recognition
The gist
Improving video quality by making videos sharper and more detailed is hard because available training videos are often limited in variety. The authors created a new, large collection of videos called SCOPE-VSR that better covers many different scenes and subjects. They also developed a new method called GraCo-VSR that uses this dataset to restore videos with fewer distortions in motion and texture. Their approach leads to clearer, more natural-looking videos.
Open → 2610.24604v1

Audio visual segmentation improves object separation with competitive slots

SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention

Abstract: Audio-Visual Segmentation (AVS) requires predicting pixel-wise masks of sounding objects by grounding acoustic signals in visual space. In multi-source scenes, concurrent sound sources often remain entangled within shared visual representations, leading to merged masks or unstable specialization when using standard alignment-focused transformer architectures. To address this, we present SlotAVS, an object-centric framework driven by competitive bi-modal slot attention. While robust foundation models provide rich semantic priors, SlotAVS shifts the paradigm from global cross-modal fusion to explicit competitive binding. By initializing discrete object slots from temporal audio sequences and enforcing strict slot-wise competition over spatial visual tokens, our method structurally mitigates the blending of distinct acoustic entities. Further, to stabilize instance-level learning, we introduce a Hungarian matching strategy that optimally aligns slots with ground-truth objects using spatial and boundary overlap. Coupled with a numerically stable, confidence-aware mask fusion mechanism, SlotAVS robustly aggregates multi-source predictions. Extensive experiments on the AVSBench dataset demonstrate the efficacy of our approach, achieving 88.6% mIoU on the Single-Source (S4) split and 71.7% mIoU on the Multi-Source (MS3) split. These results show that explicit competitive binding is critical for complex audio-visual scene understanding and effectively leveraging foundation-model priors without relying solely on their capacity.

Sun 11 OctComputer Vision and Pattern Recognition
The gist
Audio-Visual Segmentation means figuring out which parts of a video show the objects that are making sounds. The authors found that when many sound sources are present, usual methods mix the objects together or get confused. They made a new way called SlotAVS that separates the sounds and images by letting different parts of their model compete to claim objects, which keeps things clearer. Their approach helps computers better recognize and outline multiple sounding objects in videos.
Open → 2610.15764v1

Feed-forward 3D models improve depth and pose without labeled data

PURE: Refining Feed-forward 3D Reconstruction with Unlabeled Post-training

Abstract: Feed-forward 3D reconstruction models have achieved remarkable progress through large-scale training and strong learned priors. However, they remain fragile when applied to dynamic in-the-wild videos, where non-rigid motion, occlusions, and illumination changes are prevalent. A key limitation lies in their reliance on expensive depth and camera pose annotations, which hinders scaling to diverse real-world dynamic data and restricts generalization. To address this issue, we propose \textbf{PURE}, an unlabeled post-training refinement framework that improves depth and camera estimation without requiring ground-truth supervision. Instead of relying on annotations, PURE elegantly distills robust structural knowledge from state-of-the-art 2D priors into 4D feed-forward architectures. PURE leverages relative depth constraints, affine-invariant geometry distillation, and pose refinement based on feature correspondences to stabilize geometric predictions under challenging dynamic conditions. Extensive experiments on both synthetic and real-world dynamic benchmarks demonstrate that PURE delivers consistent improvements across multiple state-of-the-art architectures. Furthermore, our data-efficient paradigm effectively mitigates domain shifts, successfully unlocking the architectural limits of these base models in challenging, in-the-wild dynamic scenes.

Sat 10 OctComputer Vision and Pattern Recognition
The gist
3D reconstruction models that use fast, direct methods can struggle to understand videos with moving people, changing light, and objects blocking the view. The authors propose PURE, a way to make these models better after training, without needing any special labels like depth or camera angles. PURE uses clever clues from 2D information and relationships in the video to improve the 3D understanding of scenes. Their tests show that PURE helps many leading models work more reliably on real-world dynamic videos.
Open → 2610.14698v1

Streaming video models struggle with fast real-world action recognition

FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?

Abstract: Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.

Thu 8 OctComputer Vision and Pattern RecognitionComputation and Language
The gist
Understanding videos that have fast-moving events is challenging for current video language models, especially when they only see a few frames per second. The authors created FastBench, a new test set with carefully verified questions to check how well these models understand quick actions in real videos. They found that even the best models score around half right, improving when they get more frames but still far from perfect. A new technique called ProactiveFrame helps a bit by choosing when to look more closely, but models still can’t reliably decide when to focus on details from the video alone. This work highlights the difficulty of analyzing fast video streams and provides a benchmark for improvement.
Open → 2610.12427v1

Improved multidimensional signal modeling with dual neural networks and wavelet fusion

Spatial-Frequency-Aware Implicit Neural Representation of Multidimensional Signals via MLP-KAN Fusion

Abstract: Implicit Neural Representations (INRs) have emerged as a compelling paradigm for modeling multidimensional signals by mapping continuous coordinates to signal values. However, Multi-Layer Perceptrons (MLP)-based INRs inherently suffer from spectral bias, which favors low-frequency components and suppresses the reconstruction of essential high-frequency details. While existing techniques, such as Fourier feature mappings, mitigate this issue, they often rely on sensitive manual tuning and are prone to spectral artifacts. In this paper, we propose a spatial-frequency-aware INR framework that combines an MLP branch with a Kolmogorov-Arnold network (KAN) branch for complementary frequency-oriented modeling. The MLP branch provides a low-frequency-oriented representation of smooth structures, whereas the KAN branch complements localized variations and fine details. To coordinate the two branches, we integrate the discrete wavelet transform (DWT) and inverse discrete wavelet transform (IDWT) into the output fusion stage. The outputs of the two branches are decomposed into wavelet coefficients, and the corresponding coefficients are additively fused before inverse wavelet reconstruction. A wavelet-domain band-separation regularization further penalizes high-frequency responses in the MLP branch and low-frequency responses in the KAN branch, thereby encouraging complementary frequency-oriented behavior. Experiments on 1D signals, 2D images, 3D volumes and signed distance functions, videos, and 4D light-fields demonstrate the applicability of the proposed representation across the evaluated signal modalities. Results demonstrate improved reconstruction fidelity across the evaluated signal modalities.

Thu 8 OctComputer Vision and Pattern Recognition
The gist
Many computer techniques struggle to capture both smooth and detailed parts of images or sounds, often missing fine details. This paper introduces a method that uses two types of neural networks working together: one focuses on the smooth parts and the other on the fine details. They combine their outputs using a mathematical tool called wavelets to capture all important features. The method works well on many types of data, including images, videos, and 3D shapes, making the results clearer and more accurate.
Open → 2610.11296v1

Flow reversal improves temporal generation quality with conditional sources

Learning Conditional Source Distribution via Flow Reversal for Temporal Flow Matching

Abstract: We introduce CNP-Flow, a flow matching framework for temporal generation that learns conditional source distributions through flow reversal. Whereas standard conditional flow matching (FM) incorporates conditioning through the vector field and draws source samples from a standard Gaussian, CNP-Flow uses a conditional noise predictor (CNP) to produce an isotropic Gaussian source for each temporal condition. The CNP is supervised by source samples obtained through flow reversal, which maps observed targets backward through a pretrained FM model. A three-stage pipeline pretrains the FM model, trains the CNP, and fine-tunes the FM model using the learned source distribution, while preserving the FM backbone architecture. Across video prediction, video interpolation, and 7-DoF Franka robot motion planning, CNP-Flow consistently improves generation quality. It also matches baseline performance with fewer function evaluations. Project page: https://embodiedai-ntu.github.io/cnpflow

Sun 4 OctComputer Vision and Pattern Recognition
The gist
Generating sequences like videos or robot motions often requires simulating changes over time, starting from random noise. The authors present a way to better choose this starting noise based on the specific point in time, by learning it backward from the output. This approach improves quality and efficiency when creating videos or planning robot motions. They show it works well in several tasks involving video prediction, interpolation, and robot control.
Open → 2610.05349v1

Video diffusion transformers run faster with smarter attention retrieval

PARK: Accurate Block Retrieval for Sparse Attention in Video Diffusion Transformers

Abstract: Diffusion Transformers (DiTs) have become a dominant architecture for video generation, but their efficiency is limited by the quadratic complexity of full attention. Sparse attention reduces this cost by retrieving important blocks and computing attention only within them, but inaccurate retrieval can either degrade generation quality or yield unnecessary computation. We identify two retrieval mismatches in methods that retrieve blocks using the averaged representations of query and key blocks: (i) query-side aggregation mismatch, where averaging queries before Softmax fails to preserve their individual attention preferences, and (ii) key-side clustering metric mismatch, where standard Euclidean clustering in the original key space can group keys with dissimilar QK scores under the current query, so their average representation may not accurately represent how the current query scores individual keys. These mismatches can lead to inaccurate block retrieval. To address these mismatches, we propose PARK, a training-free sparse attention method for accurate block retrieval. PARK retains every original query, independently normalizes its attention over key blocks, and then averages these distributions within each query block. It also uses information from the current queries to transform keys before clustering, so that keys receiving similar QK scores are grouped together. A fused GPU kernel further reduces the overhead of block retrieval. Experiments on HunyuanVideo and Wan demonstrate that PARK improves block retrieval accuracy and preserves generation quality while accelerating inference, achieving the best quality-efficiency trade-off among the compared sparse attention methods.

Wed 30 SeptComputer Vision and Pattern Recognition
The gist
Processing videos using transformers is slow because the method looks at every part of the video in detail, which takes a long time. The authors found that grouping video parts for this process often mixes up important details, which hurts quality or wastes effort. They designed PARK, a way to group parts more carefully by keeping the unique information from each query and grouping keys based on how well they actually match. This method speeds up video generation while keeping good video quality without needing extra training.
Open → 2609.38978v1

Salt++ improves few-step streaming video and audio generation quality

Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation

Abstract: Few-step streaming audio--video generation requires both causal modeling and step distillation, yet standard training recipes face two context-related challenges. Teacher forcing pairs clean history with a noisy target, but supervises predictive contextual representations only indirectly through velocity prediction. Meanwhile, directly reusing bidirectional score models in causal Distribution Matching Distillation (DMD) creates a mismatch between generation and scoring contexts. We address these challenges with Salt++, a two-stage post-training framework comprising Causal Self-Flow (CSF) and context-aligned autoregressive DMD. CSF exploits contextual information asymmetry by varying the history while keeping the noisy target fixed: a noise-mixed-history student aligns its intermediate representations with those of a clean-history exponential-moving-average teacher. This self-supervised signal encourages the student to extract semantic information and improves cross-modal alignment. Context-aligned AR DMD shares the causal mask and prefix across generator sampling, fake-score training, and real-score evaluation to match generated and reference distributions under a block-conditional KL objective. With calibrated teacher guidance, it performs clean-prefix few-step distillation and then adapts to generated histories without switching objectives or requiring separate consistency distillation. At 480p, Salt++ improves visual and motion quality by 57% and 45% over OmniForcing on JavisBench under the same 4-step causal setting. A separate scale-wise post-training stage extends Salt++ to 4-step $1664\times960$ generation, outperforming bidirectional LTX-2 on six of seven reported metrics. Project page: https://xingtongge.github.io/Saltpp

Tue 29 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Generating audio and video in a few quick steps is hard because the system needs to understand both past and future information differently. The authors present Salt++, a new approach that trains models to better handle these challenges by aligning how the system learns from past context and noisy new data. This makes the generation process more accurate and better aligned across sound and visuals, improving quality significantly. They tested Salt++ on video benchmarks and showed it works better than previous methods.
Open → 2609.36995v1

Octree method speeds video processing with fewer data points

Octree-based Video Representation

Abstract: Video models commonly use uniform grids even though visual complexity varies substantially across space and time. We introduce OctVideo, which approximates a video clip with an octree. This hierarchy recursively partitions a spatio-temporal volume into eight subvolumes, so that smooth regions remain coarse while detailed regions receive finer cells. Each leaf stores local RGB values and spatio-temporal gradients, supplemented by a lightweight learned residual. For reconstruction, a Conv1D VAE maps the serialized cells to a regular latent grid and selectively refines details during decoding. Our VAE achieves 36.12 dB PSNR with 38.2M parameters and 189.4 GFLOPs per clip on Kinetics-400 (K400). It also generalizes zero-shot to the high-resolution Densely Annotated VIdeo Segmentation (DAVIS) 2016 dataset with reconstruction quality comparable to the best evaluated models. On both datasets, it requires the fewest model FLOPs and achieves the fastest encoding and decoding among the evaluated models. OctVideo also supports video understanding, achieving competitive recognition performance with few input tokens when trained from scratch. By exploiting the redundancy already present in video signals and efficiently processing sparse structures, OctVideo provides an efficient representation for video.

Sun 27 SeptComputer Vision and Pattern Recognition
The gist
Videos have lots of repeated and simple parts mixed with detailed sections, but current methods treat all parts the same, making processing inefficient. The authors introduce OctVideo, a way to represent video using a tree-like structure that breaks up space and time so simple areas are stored coarsely and detailed parts finely. This method uses fewer computations, encodes and decodes faster, and still produces good quality video reconstructions. It also works well on different video datasets without extra training and helps recognize video content using fewer inputs.
Open → 2609.33100v1

Video sparse attention speeds up generation with fine-grained routing

Improving Video Sparse Attention with Fine-grained Router and Sparse Rebasing

Abstract: We present VSA2, a frontier trainable sparse attention for video DiTs. VSA2 includes a variety of new architectural features and training procedures that we apply across all stages of the DiT development cycle, including pretraining, RL, and inference, to produce a DiT with comparable or better quality than a full attention counterpart. Architecturally, VSA2 introduces a fine-grained router that improves the precision of identifying critical tokens and supports dynamic computation by allowing each query to attend to a variable number of key-value pairs. In training, we identify a Hard-to-Easy Curriculum, where models trained under high sparsity and later evaluated with lower sparsity during inference not only generalize effectively, but also outperform models trained with full attention in motion quality. VSA2 is also flexible: it can replace full attention during the middle of progressive low-to-high resolution pretraining, rebasing early-stage full-attention checkpoints. Experiments show that VSA2 reduces attention computation by half over VSA with lower loss. On 720p videos, it accelerates attention by 8.9x and end-to-end generation by 4.62x compared to the FlashAttention-3 baseline, while achieving comparable or better video quality.

Sat 26 SeptComputer Vision and Pattern Recognition
The gist
Processing videos efficiently is challenging because it requires lots of computations to focus on important parts. The authors developed VSA2, a method that smartly picks which parts of the video to pay attention to, using a ‘fine-grained router’ that adjusts how much is processed for each part. They also found a special training approach that helps models perform even better when they focus less, making video generation faster without losing quality. Their method can be used during different training stages and speeds up video creation significantly compared to previous techniques.
Open → 2609.32882v1

Space improves video memory by predicting future use patterns

SPACE: Sparse Predictive Attractor via Counterfactual Eviction for Streaming Video Memory

Abstract: Fixed-capacity streaming video memory requires repeated eviction decisions whose effects accumulate over time. Yet existing policies are evaluated primarily in terms of retained information or downstream accuracy, leaving how repeated updates alter the futures supported by memory largely unexamined. We define a memory's predictive state as the future representations supported by its retained history and formulate eviction as counterfactual control over transitions in this space. We introduce SPACE (Sparse Predictive Attractor via Counterfactual Eviction), which uses a frozen multi-horizon JEPA to predict the future representations induced by alternative eviction actions. Counterfactual utility identifies future-useful alternatives, while slow predictive-basin geometry determines when to correct avoidable drift and when to adapt to sustained predictive change, without online parameter updates. We further introduce MABS-Bench, which evaluates future-task sufficiency, within-regime predictive stability, transition responsiveness, and perturbation recovery under matched causal streams and memory budgets. Across multiple video datasets, SPACE yields consistent improvements in dataset-native task performance while reducing predictive-state drift.

Sat 26 SeptComputer Vision and Pattern Recognition
The gist
When computers watch videos and remember what they see over time, they have limited memory and need to decide what to keep or forget. The authors propose a way to predict how different choices about what to remember affect future video understanding. Their method, called SPACE, uses a special model to foresee future video parts that will be useful and makes smarter decisions about what to keep in memory. This helps keep important information longer and improves overall video performance without needing to relearn during use.
Open → 2609.32592v1

KeyRec reduces memory use for understanding long videos and streams

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Abstract: Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose \textbf{KeyRec}, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.

Sat 26 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Videos and streams are long, and computers often slow down trying to watch all the details at once. The authors designed KeyRec, a system that keeps important recent moments in short-term memory and groups earlier events in a smart way to save space. It decides how much attention to give to recent versus older parts based on the question asked, without rewatching old video parts. This makes video understanding faster and more efficient without losing important details.
Open → 2609.32182v1

Dual-stream method improves hdr video with alternating exposures

Double-stream registration with pyramid fusion for HDR video with alternating exposures

Abstract: High dynamic range (HDR) video reconstruction from al\-ter\-na\-ting-exposure sequences remains challenging, especially in regions with extreme luminance variation. We propose a novel HDR reconstruction framework based on dual-stream registration and accurate pyramid fusion. Given three consecutive frames, our method computes optical flow directly with the central frame, while introducing a complementary midpoint displacement strategy to handle cases with severe overexposition. A pyramid fusion stage then merges the resulting radiance and LDR images into a final HDR output. Experimental results demonstrate that our approach consistently outperforms state-of-the-art methods.

Fri 25 SeptComputer Vision and Pattern Recognition
The gist
Capturing videos with very bright and very dark areas is hard because normal cameras can’t see all lighting details at once. The authors propose a new way to combine video frames taken with different brightness settings to create clearer, high dynamic range (HDR) videos. Their method aligns video frames more accurately and merges the information using a pyramid fusion technique, handling even very bright spots well. Tests show their approach works better than existing methods.
Open → 2609.31108v1