Vision language models can get worse on some tasks as they grow larger
On the Limits of Scale: Inverse Scaling in Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Usually, bigger AI models perform better, but this paper finds that for vision-language models, larger versions sometimes do worse on certain question types. The authors tested many models and found specific tasks, like multi-image reasoning and university STEM questions, where smaller models answered correctly but bigger ones failed. This happens across different model types and datasets, suggesting the problem is about the tasks, not just the models. Their work shows that choosing a model should consider the particular tasks it will do, not just assume bigger is better.
What this means in practice
- •For ai system builders: Choose vision-language models based on specific task performance rather than just size to avoid unexpected accuracy drops.
- •For educational technology developers: Design STEM-focused AI tools by selecting models that maintain performance on university-level knowledge tasks despite model size.
Authors
Rohit Saxena, Alessandro Suglia, Marek Rei, Pasquale Minervini
Abstract
Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures. We identify 13 subtask-dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific. The most affected domains in our study include multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases, while diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and suggest that model selection should be task-aware.