Video quality assessment improves by focusing on key spatio temporal patches
High-Fidelity Video Quality Assessment with VQA-Specific Saliency
Computer Vision and Pattern Recognition
Summary
Evaluating video quality without needing a perfect reference is hard because videos have lots of data and details that matter. The authors propose a new way to analyze videos by breaking them into smaller space and time patches, focusing only on the most important parts to human perception. They teach a small helper network to learn which parts really affect how we see video quality, so the system can ignore less important patches. This method keeps important details at the original video resolution while reducing the amount of data processed, making it faster and more accurate.
What this means in practice
- •For video streaming services: Assess video quality efficiently by focusing processing on important spatio-temporal video patches to improve quality monitoring without full video analysis.$Commercial implications: Enables streaming services to sell better quality control and adaptive streaming by efficiently detecting quality degradation.
- •For video editing software developers: Incorporate efficient quality assessment that prioritizes visually important video areas to speed up video enhancement and editing workflows.
Authors
Hakan Emre Gedik, Shashank Gupta, Alan Bovik
Abstract
No-reference video quality assessment (NR VQA) has recently seen promising progress with deep learning. However, video data is inherently large, and processing them with deep models incurs high computational cost. This challenge is particularly acute in VQA, where preserving original-resolution cues and dense temporal information is critical for accuracy. Existing efficiency-driven preprocessing strategies, such as fragmenting, reduce computation but alter the input data distribution, limiting effective reuse of pretrained video foundation models (ViFMs). To address these challenges, we propose \textbf{H}igh-\textbf{F}idelity \textbf{V}ideo \textbf{Q}uality \textbf{A}ssessment (\textbf{HFVQA}), a framework built on fixed-size spatio-temporal (ST) patches that is fully compatible with pretrained ViFMs. HFVQA samples ST patches across multiple scales, including the original resolution, with minimal temporal subsampling to preserve low-level quality cues and semantic context. To limit computation, HFVQA introduces a lightweight auxiliary network trained end-to-end with the ViFM encoder to learn \textit{VQA-specific saliency}. Distilled directly from quality supervision, this saliency captures task-specific importance patterns, reflecting that video quality perception is dominated by a small subset of spatio-temporal regions. By combining high-fidelity spatio-temporal cues with learned, task-specific saliency, HFVQA achieves SOTA performance on standard NR VQA benchmarks while processing as little as 12\% of candidate ST patches, making high-fidelity ViFM-based VQA computationally tractable.