CoVisco reduces visual tokens for efficient image and video understanding
CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Videos and images have a lot of visual details that make it hard for computers to understand them quickly, especially for long videos. The authors introduce CoVisco, a new way to compress and process visual information natively while the computer is still encoding it. CoVisco represents parts of a video with fewer, more abstract tokens that still keep essential details, allowing computers to understand images and videos faster without losing accuracy. This method also lets the computer choose when to look at finer details if needed.
What this means in practice
- •For video platform engineers: Create more efficient video understanding pipelines that handle long videos using compressed visual tokens without sacrificing accuracy.
- •For mobile app developers: Build apps that analyze images and short videos quickly by using compact visual token representations to reduce computation and memory.
Authors
Yulong Liu, Xiaotian Han, Junyuan Shang, Yuchen Ding, Zhenyu Zhang, Shuohuan Wang, Guibo Zhu, Sirui Han, Dianhai Yu
Abstract
Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding expensive for the vision encoder and the language model. Existing methods often compress visual tokens after dense encoding, creating a mismatch between the representation used during training and the compact interface required at deployment. We present CoVisco, a codec-native vision encoder with native token compression for unified image-video understanding. By combining codec-native input support with segmented attention, CoVisco can encode long visual inputs in a single forward pass without forming dense patch-to-patch interactions across all frames. Each temporal segment is equipped with learnable abstract tokens that learn a compact segment-level representation, while fine-grained patch tokens remain available throughout the encoder. Alternating intra-segment and abstract-communication layers preserve video-level context through the abstract-token channel. A lightweight selector further exposes either abstract tokens alone or abstract tokens augmented with a runtime-selected subset of patch tokens, yielding a compact visual interface that reduces the visual context and prefill burden of downstream MLLMs while retaining fine-grained evidence when needed. Pretrained with contrastive objectives on 565M image--text pairs and 6.4M videos, CoVisco shows competitive performance on video-oriented embedding and multimodal understanding benchmarks. In the evaluated four-segment, 64-frame setting, abstract-only inference uses only 400 visual tokens while achieving video-understanding performance close to, and on some benchmarks exceeding, OneVision-Encoder. Selected patch tokens further improve fine-grained video reasoning. Project URL: https://github.com/ernie-research/CoVisco.git