Ultra: Unsupervised Cross-Task Optimization for Reliable Restoration Segmentation Collaboration under Adverse Weather
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of teaching a computer to recognize objects in bad weather when it only has clear weather images labeled. They show that previous methods trying to fix images and segment them together can cause more mistakes when the bad weather is very severe. Their new approach, called Ultra, helps the two tasks work better by carefully choosing which directions to learn from and checking if the fixes really help. Experiments show Ultra improves both recognizing objects and restoring images without needing labeled bad weather data.
Unsupervised Domain AdaptationSemantic SegmentationAdverse WeatherImage RestorationCross-task OptimizationCausal Effect EstimationError PropagationCooperative LearningUnsupervised LearningComputer Vision
Authors
Shiqin Wang, Zhiqian Li, Haoyuan Du, Junming Chen, Jiayuan Li, Tianrun Xu, Haoyang Chen
Abstract
Unsupervised Domain Adaptation for Adverse Weather Semantic Segmentation (UDA-ASS) aims to transfer semantic knowledge from labeled normal-weather images to unlabeled adverse environments. Existing approaches implicitly assume that restoration and segmentation provide mutually beneficial guidance. However, under severe degradation and without target-domain supervision, the validity of cross-task optimization directions becomes fundamentally unidentifiable, leading to hallucination-driven error propagation. In this work, we propose a novel Unsupervised Restoration-Segmentation Collaborative Learning Framework (Ultra), which reframes cross-task interaction as direction selection under uncertainty and causal effect estimation, enabling reliable collaboration through candidate direction generation and intervention-based filtering. In detail, we propose CTDN and CMIL. The former exploits complementary visual structures and semantic information to generate candidate optimization directions and performs cooperative direction selection between restoration and segmentation. The latter reformulates cross-task information transfer from correlation-based propagation into causal effect assessment, suppressing hallucination propagation. Extensive experiments on three widely used UDA-ASS benchmarks demonstrate state-of-the-art segmentation performance. Beyond segmentation, our framework achieves better unsupervised restoration results than existing UDA-ASS restoration methods and generalizes to unsupervised restoration and object detection collaboration tasks. Code and models will be available at https://github.com/Wang-Shiqin/Ultra.