VisionWeave boosts multimodal model efficiency by adaptive visual processing
VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Large multimodal language models often process images by splitting them into equally sized pieces, which can be wasteful because some parts of an image need more detail than others. The authors introduce VisionWeave, a new approach that lets these models decide automatically how detailed each image part should be, saving computing resources without losing accuracy. They trained this method on big models and showed it reduces workload while keeping nearly the same performance. This can make systems that understand images and text faster and more efficient.
What this means in practice
- •For ai infrastructure teams: Improve server throughput and reduce latency for multimodal AI services by using VisionWeave to adaptively reduce visual token processing without major performance loss.
- •For video analysis developers: Process video frames more efficiently by allocating computational resources dynamically to important visual regions, achieving faster frame processing with maintained accuracy.
Authors
Yuan Feng, Qize Yang, Ruizhe Chen, Sibo Song, Haolin He, Muzhi Zhu, Zihan Liu, Yunfei Chu, Xize Cheng, Yuxuan Wang, Jin Xu, Xike Xie
Abstract
Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.