A Controlled Evaluation of Model Rankings and Input Reliance in Surface Water Segmentation
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors looked at how to better understand why some computer models perform better at finding surface water from images. Instead of just comparing overall scores, they tested different setups multiple times, looked at specific data parts, and checked how much the models depend on certain inputs like terrain. They found that simple rankings don't always explain which parts help most or how stable the results are across areas or tests. Their work shows that you need more detailed analysis to truly understand model performance beyond just summary numbers.
surface-water segmentationintersection-over-union (IoU)Sen1Floods11Swin-UNetU-Netancillary inputsterrain datamodel evaluationgeographic weighting
Authors
Kittipat Phunjanna, Kristóf Karacs, Chayut Ngamkhanong
Abstract
Performance evaluation for surface-water segmentation commonly uses an aggregate metric such as global intersection-over-union (IoU) to rank model configurations. However, a configuration ranking does not by itself establish why one system performs better, whether a close ordering is stable, or how strongly predictions rely on individual inputs. We examine these distinctions primarily on Sen1Floods11 through repeated configuration comparisons, paired test-chip analysis, fixed-checkpoint input stress tests, and geographic reweighting, with a targeted secondary evaluation of supervised input configurations on GEOID-Flood. The cross-modal student achieves the highest three-seed mean IoU on Sen1Floods11, but close orderings vary across seeds and geographic weighting, while ancillary-input rankings differ between Swin-UNet and U-Net. The GEOID-Flood evaluation shows substantial agreement in supervised ancillary-input effects, although the exact architecture ordering remains configuration dependent. Fixed-checkpoint tests further establish reliance on terrain and WorldCover without establishing a clean-input performance benefit, while target semantics and the later WorldCover prior restrict the evaluation to retrospective all-water segmentation. These results show that aggregate metrics remain useful for ranking complete configurations, but ranking stability, component attribution, input reliance, and deployment scope require distinct evidence. Performance evaluation should therefore match the evidence reported to the claim being made.