Joint multi-task learning improves detection across domains
JMT-YOLO: Multi-task Domain Adaptation via Segmentation-Guided Consistency Learning
Computer Vision and Pattern Recognition
Summary
Object detection models often perform poorly when tested on new types of images different from their training images, like in bad weather or different styles. The authors propose JMT-YOLO, a method that teaches the detector using both object detection and image segmentation tasks together. It also uses a technique where parts of the image are randomly hidden to encourage the model to learn the overall scene better. Their experiments show this approach works better than existing methods on several challenging domain shifts.
What this means in practice
- •For autonomous vehicle developers: Improve object detectors to better recognize vehicles and pedestrians in varied weather and lighting by adapting models trained on clear data to foggy or occluded scenes.
- •For surveillance system engineers: Enhance performance of object detection in security cameras when environmental conditions or camera types vary from training data.
Authors
Zhenhua Yu, Jiahao Zhang, Wenchao Zhang, Ou Ye, Hengxiang Zhao
Abstract
Domain adaptive object detection (DAOD) aims to reduce performance degradation due to distribution shifts between training (source) and testing (target) datasets. However, owing to insufficient modeling of fine-grained features and limited instance-level alignment, existing approaches become less effective under complex environmental conditions, such as extreme weather conditions and object occlusion. We overcome these challenges by proposing JMT-YOLO, a novel joint multi-task learning framework for DAOD. JMT-YOLO is based on the Mean Teacher architecture. It introduces an auxiliary semantic segmentation task and a masking mechanism to enhance the modeling of foreground-background semantic boundaries in the target domain. Specifically, fine-grained feature degradation is reduced by introducing a multiscale context-aware segmentation head equipped with a dense connection structure to effectively capture local and global contextual features. A mask-consistency loss is formulated by applying random masking operations to target images, thereby guiding the network to learn structural context and improving instance-level feature alignment. Extensive experiments on three benchmark domain adaptation tasks (Pascal VOC to Clipart1k, Cityscapes to Foggy Cityscapes, and KITTI to Cityscapes) demonstrate that JMT-YOLO achieves detection performances of 46.9%, 57.3%, and 61.0%, respectively. These results demonstrate superior performance over existing state-of-the-art and baseline methods, thereby validating the effectiveness and generalization of JMT-YOLO.