Transformer model layers merged to cut computing needs in half
A Smaller Transformer in Your Transformer
Computer Vision and Pattern Recognition
Summary
Some AI models that look at images use many layers that often do similar work, which can waste computer power. The authors found a way to combine these layers into fewer ones without losing much ability to understand images. Their method, called Transformer-Within-Transformer (TWT), makes the model simpler and faster while keeping it accurate. In tests on natural images and medical pictures, TWT worked as well or even better than the original models with twice as many layers.
What this means in practice
- •For computer vision engineers: Create more efficient image recognition models that run faster while maintaining accuracy by merging similar transformer layers.
- •For medical imaging teams: Deploy compact transformer models for histopathology image analysis that reduce computing costs without losing diagnostic quality.
Authors
Dhananjay Tomar, Marius Aasan, Andreas Kleppe, Adín Ramírez Rivera
Abstract
Recent findings indicate that Vision Transformers settle into locally similar computational phases, implying a level of depthwise computational redundancy. However, existing methods to exploit this redundancy either fail to reduce inference compute or severely degrade model expressivity. In this work, we formalise a unified view of block redundancy that decouples the geometry from specific surrogate interventions. We then introduce Transformer-Within-Transformer (TWT), a post-hoc method that fuses contiguous groups of redundant layers into a single learned surrogate layer. TWT reduces parameter count and inference compute while remaining competitive with original models using half the depth on natural images, and in several downstream histopathology settings, TWT matches or even improves on the original baseline.