Merging vision transformer and state space models boosts image classification
Riemannian--Lorentz Fusion of Vision Transformers and State-Space Models
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageMultimedia
Summary
Training big AI models can be expensive and slow, especially when combining different model types. The authors created a new method to blend two different vision models—a Vision Transformer and a State-Space Model—by aligning their parts in a special geometric space. This combined model performs better on several image recognition tasks after some additional training. However, the method still requires training after merging and doesn't work as a simple, training-free blend.
What this means in practice
- •For machine learning engineers: Improve image classification performance by merging vision transformers with state-space models using geometry-aware parameter fusion.
- •For computer vision developers: Build hybrid image recognition systems combining diverse model types to enhance accuracy on standard datasets.
Authors
Badri N. Patro, Vijay S. Agneeswaran
Abstract
Scaling deep learning faces critical bottlenecks: data exhaustion, exponential training costs, and resource concentration. Model merging combines pre-trained checkpoints without gradient descent, offering orders-of-magnitude savings versus retraining. Combining independently trained vision models is difficult when their architectures and parameter shapes differ. Existing weight-space merging methods generally assume aligned, shape-compatible checkpoints, whereas a Vision Transformer (ViT) and a state-space model (SSM) implement token mixing with different operators. We study a hybrid Heterogeneous merging setting that retains both architectures while aligning parameter groups by semantic role. Our proposed Riemannian--Lorentz Parameter Fusion (RLPF) method projects aligned groups to common coordinates, lifts selected coordinates to the Lorentz hyperboloid model of hyperbolic space, computes a regularized geodesic barycenter, and decodes the result into the two branches. A learned gate then combines branch logits for each input. Component groups use fixed curvature values, with normalization parameters treated as Euclidean. In the results available in this manuscript, the fine-tuned system obtains 82.37\% on CIFAR-10, 75.04\% on Oxford-IIIT Pet, and 78.58\% top-1 accuracy on ImageNet-1K; the corresponding best-parent accuracies are 76.54\%, 71.42\%, and 76.42\%. On ImageNet-1K, the reported pre-fine-tuning initialization reaches 77.80\%. These results support further study of geometry-aware heterogeneous fusion, but not a training-free single-checkpoint merge: RLPF is a two-branch hybrid whose gate and reported final models are trained.