Video semantic fidelity guides bitrate selection for better streaming
SemABR: Measuring Video Semantic Fidelity with Multimodal LLMs for Adaptive Bitrate Streaming
MultimediaNetworking and Internet Architecture
Summary
Current methods for measuring video quality mainly look at picture clarity, not whether the important story and actions in the video are still clear after compression. The authors create a new way to check if a compressed video keeps the important content intact by using language models to compare descriptions of the original and compressed videos. They then use these comparisons to choose the best video quality for streaming, especially over 5G, so viewers see videos with less loss in meaning, even when internet speed is limited.
What this means in practice
- •For video streaming engineers: Select video bitrates dynamically to preserve meaningful content during streaming, improving viewer experience when network capacity fluctuates.
- •For mobile network operators: Allocate wireless resources in 5G networks to reduce semantic content loss when streaming to multiple users under constrained bandwidth.
Authors
Shiqi Xu, Soung Chang Liew, Yuyang Du
Abstract
Conventional video metrics such as PSNR, SSIM, and VMAF measure visual distortion or perceptual quality, but they do not directly capture semantic preservation: whether compression retains a video's objects, actions, and temporal narrative. Existing Quality-of-Experience (QoE)-driven bitrate-selection and resource-allocation methods primarily aim to minimize rebuffering and bitrate switching while maximizing perceptual video quality, without explicitly considering semantic preservation. To address this gap, we introduce video semantic fidelity (SF), a metric that quantifies how well a compressed video preserves the semantic content of its source. An offline multimodal large language model (MLLM) generates structured descriptions of the reference and compressed versions of the video, and a separate text-only large language model (LLM) evaluates their semantic correspondence. The resulting content-dependent SF--bitrate profiles are cached and queried by the online bitrate selector without invoking MLLMs at runtime. Evaluations on three subjective QoE benchmarks show a consistent positive association between SF and mean opinion scores (MOS). A separate human semantic-rating study evaluates semantic preservation and shows that SF correlates more strongly with human judgments than conventional video metrics. We then embed these profiles into a 5G MEC-assisted video-on-demand (VoD) resource-allocation framework at the base station. When wireless resources cannot support high bitrate levels for all users, the framework uses the SF--bitrate profiles to select bitrate levels jointly across users and reduce the semantic loss caused by the required bitrate reductions. NS-3 simulations with the 5G NR module show that the proposed framework achieves higher average and worst-user SF than the evaluated baselines, with a widening advantage as the wireless resources available to each user decrease.