MaskMerge improves instance segmentation with smarter token merging
MaskMerge: Structure-aware Token Merging for Efficient Instance Segmentation
Computer Vision and Pattern Recognition
Summary
Instance segmentation is about identifying and separating individual objects in images, even when they look very similar. The authors found that current methods for combining parts of an image can mistakenly merge different objects if they look alike. MaskMerge fixes this by using rough object shape predictions to keep the boundaries clear while merging, which makes the process faster and more accurate. They tested their method on several challenging datasets and showed it works better than previous approaches, especially when objects look alike.
What this means in practice
- •For computer vision engineers: Improve instance segmentation models to be more efficient and accurate when objects have similar appearance.
- •For remote sensing analysts: Enhance processing of aerial images with many similar objects by reducing computation while preserving object boundaries.
Authors
Bavesh Balaji, Letian Jiang, Alexander Wong, David Clausi, Sirisha Rambhatla
Abstract
Token merging improves the efficiency of Vision Transformers by combining redundant tokens, but existing methods largely rely on feature similarity to determine which tokens to merge. This criterion is effective for semantic segmentation, yet can fail in instance segmentation, where adjacent objects of the same class often have highly similar features. We identify this failure mode as cross-instance token fusion. We introduce MaskMerge, an instance-aware token merging method that uses coarse intermediate mask predictions to constrain feature-based merging and preserve instance boundaries. We evaluate MaskMerge on two diverse instance segmentation domains characterized by high inter-instance visual similarity: PanNuke and iSAID. Against the SOTA token merging method ALGM, MaskMerge improves mAP from 30.1 to 32.1 on PanNuke while reducing computation from 229 to 153 GFLOPs and running 1.18x faster, and from 33.1 to 36.2 on iSAID while reducing computation from 198 to 182 GFLOPs and running 1.23x faster. Additionally, we also test it on Cityscapes and match ALGM's performance while using fewer FLOPs. The gains also transfer to an MAE pretrained backbone, demonstrating that instance-aware merging enables more aggressive compression when appearance similarity alone cannot reliably preserve object identity.