Adaptive frame filtering reduces video analysis cost and boosts accuracy
AdaFrame: Hierarchical Multimodal Deduplication with Adaptive Information Budgeting for Cost-Efficient Video Advertisement Analysis
Multimedia
Summary
Video analysis often processes too many similar images, which wastes computer resources without improving results. The authors propose AdaFrame, a method that smartly removes redundant frames before analyzing videos, saving lots of work while keeping the important information. They use progressive filters and measure how complex each video is to decide how many frames to keep. Tests on video ads show AdaFrame cuts the frames and computing costs significantly while improving accuracy on determining topics and sentiment.
What this means in practice
- •For video content analysts: Reduce computing costs and increase accuracy when analyzing large video datasets by removing redundant frames efficiently.
- •For advertisement technology teams: Improve topic and sentiment extraction from video ads with adaptive frame selection that balances cost and fidelity.
Tested on one dataset.
Authors
Abdul Basit Tonmoy, Arman Luthra, Guang Lin, Qixin Deng
Abstract
Uniform frame sampling for vision-language model (VLM)-based video analysis often floods multimodal backbones with redundant content, while existing keyframe selection methods are typically optimized for human viewing rather than downstream extraction accuracy. We propose AdaFrame, a content-adaptive preprocessing cascade that reduces visual redundancy without sacrificing extraction fidelity. Three progressively finer filters, perceptual hashing, learned perceptual image patch similarity (LPIPS), and CLIP-based semantic clustering, efficiently eliminate redundant frames before costly cross-modal inference. Per-video frame budgets are determined using Intrinsic Semantic Dimensionality (ISD), which estimates each video's semantic complexity through singular value decomposition of its frame embeddings. Experiments on 500 video advertisements from the Pitt Ads Dataset show that AdaFrame reduces the frame count by 48 percent and VLM inference cost by 75 percent compared with uniform sampling, while achieving 86.0 percent topic accuracy, an improvement of 1.6 percentage points, and 79.4 percent sentiment accuracy, an improvement of 7.5 percentage points.