Papers for

video streaming platforms

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Image super-resolution boosted by combining pixel and semantic context

Single Image Super-Resolution via Semantic Prompt and Cross-Modal Training on Content Area

Abstract: Enhancing image quality through super-resolution is crucial in image and video processing. While contemporary deep learning models, such as CNNs and Transformers, excel in pixel-level enhancements, they often overlook the rich semantic context within images. To address this, we introduce a novel approach that synergizes pixel intensity with semantic information for superior detail recovery. Our method involves extracting semantic data, including textual descriptors and spatial positioning, and encoding this information. We also present a Cross-Modal Feature Fusion module for generating semantic feature maps via multimodal training, and a Prompt Traction Head to augment the backbone network with semantic insights, thereby refining image details. Our empirical results across various datasets validate the superiority of our approach.

Mon 9 NovComputer Vision and Pattern Recognition
The gist
Improving blurry or low-quality images is important for many uses. The paper proposes a new method that not only improves image pixels but also understands what the image shows, using text and spatial clues. This helps produce clearer and more detailed images than just focusing on pixels alone. The authors designed special parts of their system to mix these different kinds of information and trained it with multiple types of data to get better results. Their tests show their approach works better than previous methods.
Open → 2611.12025v1

Graphene improves image super-resolution with efficient feature refinement

Graphene: A Lightweight Transformer with Ising Mean-Field Refinement for Image Super-Resolution

Abstract: This paper proposes Graphene, a lightweight image super-resolution architecture with two complementary feature refinement mechanisms: Ising-model-inspired local feature refinement using an Ising Refinement Block (IRB) and stage-history-based feature refinement using a History Attention (HistAttn) module. The IRB reuses attention output features and pre-softmax attention logits to construct signed pairwise token couplings within each local window, applies unrolled mean-field updates, and refines features through a gated residual correction. These signed couplings allow alignment and opposition between latent spins to modulate local feature refinement. HistAttn attends over the current stage output and earlier representations at the same token position and adaptively blends the current stage output with a history-weighted feature aggregate without adding spatial pairwise attention. On Urban100 at $\times4$, Graphene and Graphene-light improve over IET and IET-light by 0.24 dB and 0.38 dB, respectively, with 34.3%/13.7% fewer parameters, 44.8%/44.4% fewer FLOPs, 78.4%/71.5% lower latency, and 76.0%/85.5% lower peak memory. Real-world SR comparisons further show that Graphene recovers fine details and preserves textures across diverse degradations. Code and reference checkpoints are available at https://github.com/skethz/Graphene

Sun 8 NovComputer Vision and Pattern Recognition
The gist
Image super-resolution is the problem of making low-quality images clearer and sharper. The authors propose Graphene, a lightweight system that refines image features by using ideas from physics and attention techniques to improve detail reconstruction. It uses a special block inspired by the Ising model to refine local features, combined with a history-based attention that blends current and past features. Graphene achieves better performance than previous methods with less computational cost and memory use, while producing clearer images in real-world tests.
Open → 2611.11139v1

Automated pipeline generates and verifies quiz questions from videos

Automated Video-to-QA Pipeline: End-to-End Question Generation and Verification Using Large Language Models

Abstract: The exponential growth of educational and informational video content on platforms like YouTube presents a challenge for efficient knowledge retrieval and assessment. This paper proposes an end-to-end automated pipeline for extracting structured summaries and generating high-fidelity Question-Answer (QA) pairs from video streams. The framework integrates a cascade of state-of-the-art models: audio extraction is handled via FFmpeg, followed by automatic speech recognition (ASR) using Whisper (A library in Python) to generate accurate transcripts. Due to the token constraints of Large Language Models (LLMs), transcripts are segmented and summarized using Facebook's BART. A key innovation of this work is a Self-Verification and Fine-Tuning Loop. We utilize meta-llama/Llama-3.1-8BInstruct to generate candidate QA pairs from transcript chunks, which are then fed back into the model to rigorously assess their relevance and importance. Only verified, high-quality pairs are retained to construct a synthetic dataset. This dataset is subsequently used to fine-tune the Llama model using Low-Rank Adaptation (LoRA), resulting in a specialized model capable of generating domain-aligned assessment questions. This approach demonstrates a scalable method for creating self-improving educational tools without the need for manual data annotation. The proposed model can be downloaded from Kaggle with all the codes from GitHub.

Thu 5 NovComputation and Language
The gist
Video content is growing fast, making it hard to find key information and create quizzes. The authors propose an automated system that turns videos into text, summarizes it, and then creates and checks quiz questions using advanced language models. The system improves itself by fine-tuning on verified questions without needing people to do manual labeling. This can help build tools that automatically create study materials from videos.
Open → 2611.07755v1

Latent workspaces improve long-sequence recommendations with efficient reasoning

Read More, Reason Deeper: Latent Workspaces for Long-Sequence Recommendation

Abstract: Long-sequence recommendation can improve along two axes: \emph{evidence scaling} exposes longer histories, while \emph{reasoning scaling} adds layers that interpret that evidence for a candidate. Existing models couple them: sequence encoders propagate every added event through depth, whereas one-shot selection prevents later layers from revisiting omitted evidence. We introduce the Latent Workspace Ranker (LWR), which keeps typed evidence in persistent banks and propagates a fixed-size candidate-local workspace. Each layer rereads the banks from its evolving workspace and archives compact post-interaction state, allowing history length and reasoning depth to grow independently. We compare four architectures across various history--depth configurations using evidence scaling at fixed depth, reasoning scaling at fixed 10K history, and their joint compute-efficient frontier. LWR leads all families at long histories, continues improving with reasoning compute, and converts additional training compute into quality gains at the fastest fitted rate on the joint frontier. At 10K history, it improves NE by 0.19% over HSTU while using 69.6% fewer forward GFLOP/request. It also outperforms existing baselines on two billion-scale ranking surfaces. These results show that separating addressable evidence from propagated state makes reading more and reasoning deeper independently controllable and jointly useful.

Thu 29 OctInformation Retrieval
The gist
Recommending items based on long histories is tricky because systems mix how much data they look at with how deeply they analyze it. The authors introduce a new approach called Latent Workspace Ranker that keeps evidence separate and lets the system look at more data and analyze it more deeply without slowing down too much. This separation allows recommendations to improve by using longer history and more thinking steps independently. Their method shows better results and efficiency on very large datasets compared to existing techniques.
Open → 2610.39188v1

Film character identification from dialogue and video like a human

As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

Abstract: Films introduce characters incrementally as they progress. We study the problem of identifying principal characters and assigning their names directly from the movie itself, without using external cast lists, actor photographs, or pre-built character banks. We introduce the character naming task: given a name mentioned in dialogue and the associated video context, the model must determine which visible character bears that name. This task is challenging because name resolution often depends on cinematic cues such as shot-reverse-shot structure, self-introduction, direct address, third-party reference, gaze, and temporal continuity. We provide a dataset and annotations for evaluating character naming and first character appearances, develop a video-and-dialogue model for the task, and compare it with proprietary multimodal large language models. We further introduce a progressive framework for building a character bank as the film unfolds. Together, these contributions support online, human-like movie understanding in which character identities are inferred from the film itself.

Tue 27 OctComputer Vision and Pattern Recognition
The gist
Movies introduce characters bit by bit, and figuring out who is who is tricky without extra information like cast lists. The authors study how to identify key characters and their names just from the movie's video and dialogue, similar to how humans do it. They look at cues like how people speak, where they look, and which shots follow one another. The authors created a dataset and a model that can name characters as the film unfolds, and compared it to existing language-based models. This approach helps computers understand movies in a more human-like way by recognizing characters from the film itself.
Open → 2610.37579v1

Video memory management improves streaming video understanding accuracy and efficiency

Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding

Abstract: The rapid growth of video data has created an urgent need for efficient video understanding systems that can handle long and continuous visual inputs, posing significant challenges for Multimodal Large Language Models (MLLMs) due to severe memory bottlenecks. Moreover, existing compression methods often overlook the explosive accumulation of approximation errors. To address this issue, we propose Video-MemComp, a training-free and sensitivity-aware context management framework for memory-constrained streaming video understanding. Our method theoretically decomposes compression error into orthogonal local approximation error and layer-wise structural sensitivity via a first-order Taylor expansion. To minimize local approximation error, we perform token aggregation in a pre-RoPE safe zone with an attention-conserving logit bias, mitigating geometric distortions caused by RoPE rotations. To manage layer-wise structural sensitivity, we dynamically allocate the KV cache budget based on offline gradient priors and online context entropy, prioritizing high-sensitivity layers. Video-MemComp outperforms or achieves performance comparable to full-cache and state-of-the-art compression methods on benchmarks including MLVU, VideoMME, and EgoSchema. Notably, it improves the overall accuracy of Qwen2.5-VL-3B from 62.3% to 64.1% while reducing the KV cache to an equivalent of 6,000 tokens per layer, saving over 90% of KV memory.

Fri 23 OctComputer Vision and Pattern Recognition
The gist
Handling long video streams for AI models is hard because it needs a lot of memory and can introduce errors when compressing data. The authors created Video-MemComp, which cleverly manages memory by analyzing how sensitive different parts of the video processing are to changes. Their method carefully groups frame tokens and decides where to spend memory to keep important details. This approach works better than previous methods, improving accuracy while greatly reducing memory use.
Open → 2610.32784v1

OmniGround improves timing accuracy in multimodal video and audio analysis

OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training

Abstract: Omni-modal large language models can jointly understand video, audio, and text, yet they remain weak at temporal grounding, a capability essential for real-world audio-visual applications. To understand this limitation, we look beyond final predictions and examine the model's internal temporal evidence. We identify a temporal generation gap: the internal attention of the model localizes the target moment more accurately than the timestamps it generates. To close this gap, we propose TACG (Temporal Attention-Consistency Grounding), a post-training method that treats internal attention as a reliable temporal pivot and aligns it with both the model prediction and the ground-truth annotation through attention-consistency training. We further build a 580K-sample training dataset by reconstructing temporal QA from dense audio-visual annotations and organizing general omni-modal instruction data. Using TACG and this training dataset, we train OmniGround, a general-purpose omni-modal model with stronger temporal grounding capability, without injecting any additional temporal cues. OmniGround reaches state-of-the-art omni-modal temporal grounding, including 37.5 mIoU on LongVALE and 35.7 mIoU on UnAV-100, while staying competitive on both general audio-visual understanding and vision-only grounding.

Mon 19 OctComputer Vision and Pattern Recognition
The gist
It is hard for large language models to accurately find the exact times in videos or audio where events happen. The authors found that the model's internal focus actually points to the right moments better than its final answers. They created a new training method that aligns what the model focuses on with its predictions and the correct times. They also built a large training set to help the model learn better. Their improved model, OmniGround, is better at pinpointing when things happen in videos and audio without needing extra timing clues.
Open → 2610.25929v1

LoopMoEVR restores ultra-HD videos across multiple degradation types

LoopMoEVR: Loop-Based Degradation-Aware Mixture-of-Experts for Unified UHD Video Restoration

Abstract: Recently, unified high-definition image restoration has attracted considerable attention; however, existing models tend to excessively increase their depth in pursuit of improved generalization, which often yields only limited gains. Meanwhile, loop-based learning paradigms have drawn widespread attention due to their low parameter counts and strong regression capability, as exemplified by GPT-6 and looped Transformers. In this paper, we introduce the loop learning paradigm to address restoration tasks that require cross-domain learning. Specifically, we propose LoopMoEVR, a loop-based mixture-of-experts model capable of handling degraded ultra-high-definition (UHD) inputs. First, a degradation-conditioned low-rank loop embedding is designed to construct input-dependent stage conditions. Second, a spatio-temporal iterative adaptive normalization module, termed IterAda3DN, is developed to fuse local features with global loop context, thereby performing position-wise affine modulation. Finally, the expert branches further integrate the attention-updated local and global video states with the loop conditions to generate dedicated modulation parameters, while an input-conditioned depth predictor adaptively configures the number of loop iterations. With only approximately 0.884M trainable parameters, the proposed model uniformly handles UHD video dehazing, deraining, denoising, and low-light enhancement tasks, achieving state-of-the-art restoration performance on both public benchmarks and real-world scenarios.

Sun 4 OctComputer Vision and Pattern Recognition
The gist
High-definition videos can look bad because of things like haze, rain, noise, or low light. The authors designed LoopMoEVR, a new kind of video restoration model that can fix many of these problems using a clever loop-based approach. This model uses a small number of settings to understand how the video is degraded and then improves the video quality step-by-step. It works well on ultra-high-definition videos and outperforms other current methods despite using fewer parameters.
Open → 2610.05109v1

Real-time thumbnail selection improves short video engagement globally

Billion-Scale Thumbnail Optimization for Uncurated Short-Form Videos via Multi-Armed Bandits

Abstract: This paper introduces a real-time thumbnail optimization system deployed at a global $O(B)$ scale on a major short-form video platform. Unlike traditional long-form content, where custom thumbnails are heavily curated by creators, a considerable fraction of short-form videos are published without human-selected artwork. To address this uncurated corpus, we present a fully automated, end-to-end framework that replaces static default frames with dynamic, data-driven selections across billions of videos. To the best of our knowledge, this is the first published work demonstrating an online Multi-Armed Bandit framework successfully deployed at an $O(B)$ scale for uncurated short-form video discovery. Our solution pairs a multi-stage candidate generation pipeline with a low-latency serving infrastructure. By initializing the exploration framework with image-specific priors derived from a deep visual quality model, the system minimizes exploration costs and dynamically serves optimal thumbnails at serving time. Global deployment demonstrates statistically significant improvements in core user discovery and engagement metrics.

Sun 4 OctMachine Learning
The gist
Short videos often don’t have customized thumbnail images chosen by their creators, which can make it hard for viewers to decide what to watch. The authors built a system that automatically picks better thumbnail images in real time for billions of short videos worldwide. They use a smart method that tries different images and quickly learns which ones get more viewer attention, guided by a model that predicts image quality. This approach improved important user engagement measures when used on a major video platform.
Open → 2610.04931v1

Super-resolution works better in a vision model’s latent space

Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate

Abstract: In an image latent space, the embeddings of high-resolution, natural, and sharp images form a manifold. Degradation of high-resolution images pushes their embeddings off this manifold. Real-world super-resolution (SR) then becomes the task of mapping the degraded embedding back onto this manifold --- not anywhere on the manifold, but to the point that preserves what the input still carries, both its semantics and pixel details. Every published method implements this mapping in a reconstruction-oriented latent space or pixel space. We claim these spaces are the wrong substrates for SR. Low-resolution and degraded images are embedded far from the manifold, making the mapping difficult and expensive. The lack of semantic information in these substrates also makes it difficult to navigate to the faithful point on the manifold, causing severe hallucination when degradation is heavy. Thus, restoring in a suitable latent space is crucial to the SR task. We show that the latent space of 23 fused layers of a frozen DINOv3-L is one such space that makes the SR task easier. Degraded images are embedded near the manifold. Moreover, this substrate contains a hierarchy of information, from pixel record to degradation robust semantics, guiding the model to find the faithful point on the manifold. On this substrate, a 415M decoder is trained under reconstruction and adversarial objectives to map the degraded embeddings back and decode to pixel space in one pass. The resulting model, RAESR, attains the best fidelity--perception trade-off among state-of-the-art adversarial and diffusion-based restorers on RealSR, DRealSR, LSDIR and DIV2K-Val, at 37 ms per 512 by 512 image on a single H20 GPU. Swapping the substrate for a VAE latent under an identical recipe loses on every metric.

Sat 3 OctComputer Vision and Pattern Recognition
The gist
Super-resolution is the task of making blurry, low-resolution images sharper and clearer. Existing methods try to fix these images by working in spaces that do not have enough semantic information, causing errors and fake details. The authors found that using a special hidden space inside a frozen vision model called DINOv3-L lets them represent degraded images closer to the true space of sharp images. This makes restoring images more accurate and faster. Their approach beats other modern methods on several benchmarks while running quickly.
Open → 2610.04781v1

Resarc improves ultra-low bitrate image compression with residual correction

ResARC: Residual-Aware AutoRegressive Coding for Ultra-Low Bitrate Image Compression

Abstract: Progressive autoregressive image codecs provide an appealing paradigm for generative compression by quantizing continuous latents into discrete tokens, transmitting coarse-to-fine prefix tokens and generating the remaining suffix tokens at the decoder. However, their reconstruction quality is fundamentally limited by two residuals introduced along this pipeline: the quantization residual, arising from information loss during discrete tokenization, and the generation residual, resulting from imperfect autoregressive generation of the suffix tokens. To address these limitations, we introduce ResARC, a residual-aware autoregressive codec that explicitly compensates for both residuals at the decoder. Specifically, we generate the quantization residual with a diffusion transformer conditioned on the autoregressive decoding context, while requiring no additional side information. In parallel, we compute the generation residual at the encoder and employ a learned Generation Residual Codec to efficiently compress and transmit it for decoder-side correction. The recovered residuals are then integrated with the reconstructed latent representation and decoded through an adapted VAE decoder. Extensive experiments demonstrate that ResARC achieves competitive perceptual similarity while substantially improving distributional fidelity over leading generative codecs across the ultra-low bitrate regime. Code and models will be released soon.

Wed 30 SeptComputer Vision and Pattern Recognition
The gist
Compressing images into very small file sizes often leads to quality loss because some details are lost when the image data is simplified. The paper presents ResARC, a method that helps fix these losses by predicting and correcting the missing details during decompression. It uses two steps to handle different types of errors—one to guess lost details from the data sent and another to send extra helpful information to fix mistakes. This approach helps create images that look better and match the original details more faithfully at extremely low bitrates.
Open → 2609.39451v1

Video super-resolution improves low-quality videos using local frame context

LoCoVSR: Local Context Diffusion Posterior Sampling for Video Super-Resolution

Abstract: Video super-resolution (VSR) is an ill-posed inverse problem that aims to reconstruct a high-resolution (HR) video from a noisy, low-resolution (LR) version of it. We present LoCoVSR, a diffusion-based VSR framework that leverages pixel-space denoising diffusion probabilistic models. LoCoVSR integrates the Diffusion Posterior Sampling technique with spatio-temporal context learning, operating in a moving-average form. A localized window of adjacent LR frames is used for recovering each center frame, while applying a shared noise trajectory across all frames. The localized windowing enables processing of long videos without length limitations, supports parallel inference, and prevents error accumulation that may occur in recursive processing. Unlike prior methods, LoCoVSR offers a simple yet very effective VSR solution, avoiding explicit optical flow estimation, or information loss caused by latent space processing. Trained on the VFHQ face dataset, LoCoVSR achieves accurate, temporally consistent and high-quality upscaling with competitive results against recent diffusion-based VSR approaches.

Sat 26 SeptComputer Vision and Pattern Recognition
The gist
Video super-resolution means turning low-quality videos into sharper, clearer high-quality ones, but it’s a tricky problem with many possible solutions. The authors created LoCoVSR, a new method that looks at small groups of nearby low-resolution video frames to better guess the details in the high-resolution video. Their method avoids complicated steps like estimating motion between frames and works well on long videos without losing quality over time. They tested LoCoVSR on face video data and showed it makes videos look sharper and smoother over time compared to similar recent methods.
Open → 2609.32742v1

PhoenixSR improves real-world image super-resolution efficiency and quality

PhoenixSR: Generative Heterogeneous Distillation Unleashes Efficient Models for Real-World Super-Resolution

Abstract: Real-world image super-resolution (SR) requires recovering perceptually realistic high-resolution images from complex low-resolution observations while preserving faithful content. Diffusion-based SR benefits from strong generative priors but incurs substantial computational overhead, whereas feed-forward CNN and Transformer SR models are efficient yet often struggle to recover realistic high-frequency details. This motivates a natural question: can diffusion priors be transferred to existing diffusion-free SR networks without introducing diffusion components at inference time? To this end, we propose PhoenixSR, a generative heterogeneous distillation framework that transfers diffusion priors to independently designed feed-forward SR networks through score-based distribution matching. Rather than aligning heterogeneous features or imitating sampled diffusion outputs, PhoenixSR uses the pretrained diffusion model as distribution-level supervision, while paired SR supervision preserves reconstruction fidelity. To make distribution matching effective for fidelity-sensitive SR, we introduce Heterogeneous Distribution Adaptation, which adapts the target score to the SR domain, improves tracking of the evolving student distribution, and anchors training with paired supervision. We further employ Directional Reliability Weighting, a lightweight residual-consistency-based reweighting strategy that reduces unstable distributional guidance. All diffusion-related components are removed after training, leaving the original student architecture and inference cost unchanged. Experiments on three SR benchmarks and six feed-forward backbones, including SwinIR, HAT, Real-ESRGAN, and SeeMoRe, show consistent perceptual improvements with largely preserved reconstruction fidelity.

Fri 25 SeptComputer Vision and Pattern Recognition
The gist
Enhancing low-quality images to look realistic and detailed is hard, especially in real-world situations where images can be very complex. The authors introduce PhoenixSR, a new way to teach faster, simpler image-enhancing models by learning from a slower but smarter model type called diffusion models. PhoenixSR transfers knowledge without needing the slow parts when actually improving images, keeping the process fast and efficient. Experiments show that PhoenixSR makes image results look better while still keeping details accurate.
Open → 2609.30988v1