SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors highlight that while many methods adapt what a foundation model does, the way images are broken into tokens before processing is usually fixed and not content-aware. They propose SpiralFovea, a new method that changes how images are tokenized based on the local complexity of the image, creating fewer but smarter tokens arranged in spiral patterns around important parts. This approach reduces the number of tokens and computational effort while improving accuracy on fine-grained image tasks. Their tests show that the benefits are greater for models that don’t strongly use positional information.
foundation modelstokenizationvisual entropytransformerViT (Vision Transformer)self-attentionfine-grained benchmarksself-supervised learningpositional encodingadaptive inference
Authors
Kyan Mahajan, Mohammad Saqlain
Abstract
Most adaptive-inference techniques for foundation models change what the model does - early exit, MoE routing, KV-cache compression, dynamic attention sparsity. The input that hits the backbone, however, remains a fixed-grid tokenisation indifferent to image content. We argue that this is a missed lever. We present SpiralFovea, a parameter-free, input-adaptive tokeniser in which token identity, location, scale, and count are all functions of local visual entropy and selection completes before any backbone parameter is queried. Around content-driven hotspot anchors, multi-scale spiral rings produce <= 78 patches that replace the standard 196-patch ViT grid at the input stage. Across four canonical fine-grained benchmarks, SpiralFovea yields +1.7-2.1 pp accuracy with a 60% reduction in input tokens, an 84% reduction in self-attention FLOPs at every transformer layer, and 18-29% throughput gains over the matched static tokenisation baseline. A controlled ablation on CUB-200-2011 Genus across four backbones reveals a clean diagnostic: the gain magnitude tracks inversely with the strength of the backbone's whole-image positional prior, isolating self-supervised foundation models as the regime where input-adaptive tokenisation is most valuable.