Lightweight frame compensation improves speed and accuracy in real-time polyp segmentation
Lightweight Frame Compensation for Real-Time Polyp Segmentation under Frame Skipping
Computer Vision and Pattern Recognition
Summary
Polyp segmentation in colonoscopy videos needs to be fast but usually slows down when accurate models run on every frame. The authors propose a small add-on called Lightweight Frame Compensator that combines information from three nearby video frames to fix problems caused by skipping frames. This method keeps accuracy close to analyzing every frame, while more than doubling processing speed. It works well across different advanced models and helps keep the video output stable and clear.
What this means in practice
- •For medical device developers: Integrate the Lightweight Frame Compensator to enable faster and accurate polyp segmentation in colonoscopy video systems with limited processing time.$Commercial implications: This enables production of colonoscopy tools that provide reliable real-time polyp detection without costly hardware upgrades.
- •For video analytics engineers: Apply the frame compensation module to maintain accuracy when processing video streams with skipped frames to increase throughput.
Authors
Weihao Gao
Abstract
Real-time polyp segmentation in computer-assisted colonoscopy is hampered by the difficulty of deploying high-accuracy models under strict latency budgets. Frame skipping, which performs inference every K frames and reuses or interpolates the rest, improves throughput but introduces motion blur and lesion misalignment that degrade temporal consistency. To close this gap, we propose the Lightweight Frame Compensator (LFC), a 77K-parameter plug-and-play module inserted before a frozen segmentation backbone. Operating on a three-frame window, LFC fuses consecutive frames via spatio-temporal attention into a single compensated frame, and is trained only on pseudo-labels generated by a differently seeded model, without any expert annotation. Experiments on three polyp video benchmarks show that LFC recovers most of the accuracy degraded by direct frame skipping and, in several settings, reaches the accuracy of frame-by-frame inference, while retaining a 2.05X--2.55X throughput gain. We further evaluate the temporal stability of three backbones spanning CNN, pyramid Transformer, and global-attention Transformer architectures, revealing a spectrum of sensitivity to frame skipping that arises from how locally or globally each architecture aggregates misaligned input. The compensation gain of LFC follows this spectrum: it is largest for high-capacity Transformer backbones, which achieve the best static accuracy but suffer the most under skip-frame inference. LFC thus offers a lightweight, pluggable, and deployment-friendly route to real-time polyp segmentation.