Vision language model performance depends on resolution and model size
Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models need to balance how much of the image they look at and how big their language part is to understand complex questions. The authors found a simple rule called the Separable Law that predicts how well these models do when changing image details and language size. They tested many models at different resolutions and sizes, showing that some questions get better with scaling while others do not. This rule also helps decide the best way to spend computing power for better results when working with big images and large models.
What this means in practice
- •For computer vision teams: Optimize model size and image input resolution to get the best accuracy for visual question answering tasks within fixed compute budgets.
- •For machine learning infrastructure engineers: Allocate resources efficiently by choosing the right balance of model size and input image resolution based on the Separable Law to reduce waste in deployment pipelines.
Authors
Xinye Zhao, Yunkai Dang, Yunchen Wu, Wenbin Li
Abstract
Vision-language models (VLMs) face a fixed-budget trade-off between processing more visual information for fine-grained perception and using a larger language backbone for complex reasoning. Existing studies do not tell us which combination of backbone size and input resolution to deploy, especially in high-resolution deployments. To address this gap, we propose the Separable Law that describes how VLM performance changes with language backbone size and visual token count. We fit the law to measurements from 26 InternVL and QwenVL models, with language backbone sizes from 1B to 72B, on four high-resolution benchmarks with image sizes from 224 pixels to 8K. We find that the questions responding to scaling can be predicted from the skill they require, while a substantial fraction never responds at all. We also find that the two model families gain similarly from a larger backbone, while their gains from more visual tokens differ sharply. Combined with a cost law, the Separable Law gives a closed-form rule for allocating compute between backbone size and visual tokens. When deployment is limited to available configurations, the law identifies model and image sizes that perform close to the best feasible choice under the same budget. We hope our work offers a principled way to decide how much a model should be allowed to see at high resolution, given what it must reason about.