Subtoken Vision Transformer for Fine-grained Recognition
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors introduce SubViT, a method that breaks down important parts of an image into smaller pieces, called subtokens, allowing a vision transformer to focus more on the details that matter for recognizing similar-looking objects. They train the model in two steps: first by teaching it to handle various subdivided image parts, then by using a lightweight tool to quickly pick the important parts without extra computations. Their approach improves accuracy on tasks where the model must recognize fine differences and new categories without many labels, while keeping the extra processing fast and efficient. They also show it works well on other image datasets beyond fine-grained tasks.
Vision TransformerImage TokenizationFine-grained Visual RecognitionAttention MechanismSubtokensGeneralized Category DiscoveryFeature DegradationInference LatencyKnowledge DistillationDINOv2
Authors
Jie Zhu, Ivy Zhang, Minchul Kim, Xiaoming Liu
Abstract
We present Subtoken Vision Transformer (SubViT), a selective image tokenization method for fine-grained visual recognition. Standard Vision Transformers compress each fixed-size patch into a single token, although fine-grained distinctions often depend on localized variations within only a few patches. SubViT addresses this mismatch by representing discriminative patches with multiple subtokens while retaining the original token sequence for global context, thereby allocating additional capacity where it is most needed. Since attention heads encode complementary semantics and extracting attention maps at inference requires an extra backbone forward, we adopt a two-stage training strategy. Stage 1 fine-tunes the ViT using subdivision regions sampled from random attention heads, exposing the model to diverse subdivision patterns. Stage 2 identifies informative attention maps through feature-degradation distances and distills them into a lightweight single-map router, which directly predicts deterministic token-importance scores without a separate attention forward. We evaluate SubViT on Generalized Category Discovery (GCD), a challenging task requiring both fine-grained discrimination and generalization to unlabeled novel categories. Across CUB, FGVC-Aircraft, and Stanford Cars, SubViT improves the average novel-category accuracy of DINOv2 from $81.3\%$ to $84.7\%$, with only $0.50$ ms additional latency and $3.4\%$ more FLOPs, while reducing latency by $73.8\%$ relative to Retina Patch. Results on CIFAR-10 and ImageNet-100 demonstrate its broader applicability.