Hybrid vision model improves image tasks with faster efficient design

LiAuto-MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba

Computer Vision and Pattern Recognition

Summary

Understanding images requires models to process local details and spatial relationships in multiple directions, which can be tricky for some existing methods. The authors combined techniques from convolutional neural networks, Mamba sequence models, and Transformers to build a hybrid vision model called LiAuto-MindViT. Their key innovation, the Adaptive Bidirectional Mamba, blends information from both directions to avoid bias and improve understanding. They also introduced a special module that speeds up processing without extra memory costs. Tests showed this new model performs very well on tasks like recognizing images, locating objects, and understanding image segments.

What this means in practice

  • For computer vision engineers: Build faster and more accurate models for image classification, object detection, and segmentation using the hybrid LiAuto-MindViT architecture.
  • For mobile app developers: Develop efficient vision-based mobile apps with improved inference speed and reduced memory usage via the RepConvSE module.$Commercial implications: Enables lightweight, fast vision processing modules for mobile apps demanding real-time image understanding.

Authors

Lifu Mu, Shuai Chen, Wen Zheng, Haoyi Sun, Xueyang Fu, Pengfei Yu, Ning Mao, Tao Wei, Zhou Pan

Abstract

While Mamba-based models have shown strong potential for long sequence modeling, adapting them to vision is challenging due to the requirement of local neighborhood correlations and multi-directional spatial contexts for visual understanding. In this paper, we present LiAuto-MindViT, a novel hybrid vision backbone that synergizes the strengths of CNNs, Mamba, and Transformers. The core of our design is the Adaptive Bidirectional Mamba (ABM), which eliminates the directional bias of unidirectional SSMs through bidirectional selective scanning with learnable alpha blending, enabling content-adaptive directional fusion without the overhead of exhaustive multi-path routing. To further accelerate inference, we propose a deployment-friendly Reparameterized ConvSE (RepConvSE) module that leverages structural reparameterization to reduce latency and memory access overhead. Extensive experiments demonstrate that LiAuto-MindViT achieves state-of-the-art performance on image classification, object detection, and semantic segmentation while enabling efficient inference through reparameterization.
View PDFOpen arXiv