LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine LearningPerformance
AI summaryⓘ
The authors introduce LUMA, a simple and flexible mask-transformer head that can work with any image feature extractor to perform segmentation, making it easier to compare different backbone models fairly. They show that LUMA matches the best current methods but with lower computational cost and works across various backbone architectures. Using LUMA, they test many backbones and pretraining methods on common datasets to see what really affects segmentation quality. They find that simpler transformer models perform better in speed and that the choice of pretraining method matters more than the backbone design. This helps clarify what factors truly impact image segmentation performance.
transformerimage segmentationbackbonemask-transformer headcross-attentionpretrainingADE20KCityscapesViTtoken mixer
Authors
Tobias Christian Nauen, Anosh Billimoria, Federico Raue, Stanislav Frolov, Brian B. Moser, Andreas Dengel
Abstract
Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at lower cost, while attaching unchanged to isotropic, hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K and Cityscapes under one modern recipe. We find that ``efficient'' token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto-front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs segmentation quality.