Network training improves image segmentation boundaries with crf loss
CRF Loss is How Networks Should Learn Boundaries in Weakly Supervised Segmentation
Computer Vision and Pattern Recognition
Summary
Segmenting objects in images using only labels that say what objects are present is hard, especially near the edges of those objects. The authors found that previous methods used separate steps to fix boundaries which sometimes made mistakes by forcing pixels to be definitely one thing or another. Instead, they propose a method that teaches the network to learn both soft guesses about what each pixel might be and information about where the edges likely are, all at once. This approach helps produce more accurate outlines for objects in pictures.
What this means in practice
- •For computer vision engineers: Improve object segmentation models in applications where detailed pixel-level labels are unavailable but image-level tags exist.
- •For autonomous vehicle teams: Enhance boundary accuracy for segmenting road objects using limited supervision, aiding perception systems that rely on imperfect labels.
Authors
Joshua Li, Yuri Boykov
Abstract
Weakly Supervised Semantic Segmentation (WSSS) learns pixel-level predictions from image-level tags. Recent work focuses on improving coarse CAMs extracted from large vision-language models (commonly CLIP), but does little to improve their accuracy along segment boundaries. That job is instead delegated to a post-processing method like DenseCRF. However, because DenseCRF relies on low-level colour cues, it can flip correct labels to incorrect ones when neighbouring pixels share similar colours. SAM has recently been adopted as a natural alternative, yet it simply takes on DenseCRF's role as an intermediate "refinement" step that outputs one-hot pseudo-labels in prior work. By discarding the valuable uncertainty in CAMs, these one-hot pseudo-labels turn borderline errors into confidently wrong targets. Our key insight is that CAMs should supervise training alongside SAM boundaries, each through its own loss, rather than being fused together into a single hard target. Inspired by CRF potentials, we propose a framework that disentangles soft pseudo-labels as unary supervision and binary edge maps as pairwise supervision. We realize our framework in a single-stage model, DS-CRF, using CAMs from dino.txt and boundaries from SAM. DS-CRF sets a new state-of-the-art of 56.5% mIoU on MS COCO.