Video quality assessment improved with detailed regional analysis
Explicit Regional Representations For Video Quality Assessment
Computer Vision and Pattern Recognition
Summary
Measuring video quality usually looks at the video as a whole, missing detailed clues about which parts are worse or better. The authors created a new way to show exactly where in the video quality problems happen, using something called a Quality-Relevance Heatmap (QRH). This map combines local visual issues with general visual knowledge, helping existing video quality tools focus on important areas without extra labeling work. Their experiments show this approach matches well with random methods and can improve overall accuracy when fine-tuned.
What this means in practice
- •For video platform engineers: Use regional quality maps to better detect and prioritize degraded video areas during quality evaluations, improving monitoring and user experience.
- •For video encoding developers: Focus encoding resources on video regions with predicted quality issues flagged by regional representations, potentially improving compression efficiency without losing quality.
Authors
Fuyi Yang, Lei Yang, Andrew Liu, Chenyu Wei, Xiaoyi Guo, Ming Kai Hsu
Abstract
End-to-end video quality assessment (VQA) methods leave the regional evidence underlying their predictions largely implicit. We introduce an explicit pixel-level regional representation for VQA, instantiated as a Quality-Relevance Heatmap (QRH) that combines localized degradation evidence with perceptual priors under vision-language model guidance, without requiring pixel-level annotations. We apply QRH to existing VQA models through two deterministic fragment-sampling strategies: distribution-preserving sampling approximates the original sampler's QRH-score distribution without retraining, while quality-aware sampling prioritizes regions with high QRH scores and is paired with fine-tuning. Experiments with DOVER on multiple benchmarks show that the former closely approximates predictions averaged over random samples, while the latter improves most evaluated metrics across popular VQA benchmarks. These results support explicit regional representations as a useful interface for downstream VQA.