Unified vision model improves multiple image tasks simultaneously
AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection
Computer Vision and Pattern Recognition
Summary
Vision models usually specialize in one task like detecting objects or estimating depth, but combining many tasks is hard because they need different outputs. The authors created AHMAD, a system that can handle five vision tasks together by sharing parts of the model while keeping some task-specific features. They also made keypoint detection more efficient by teaching the model to do it in one step instead of many. This approach works well across tasks like segmentation, detection, and depth estimation, saving time and resources.
What this means in practice
- •For computer vision engineers: Improve resource use by deploying one model for multiple image analysis tasks including segmentation, detection, and depth estimation.
- •For mobile app developers: Run efficient pose estimation on whole images in one step, reducing computation for apps that track human movement.
Authors
Mohammad Mahdi, Nedyalko Prisadnikov, Yuqian Fu, Carmelo Scribano, Danda Pani Paudel, Luc Van Gool
Abstract
Generalist multitasking vision models aim to unify multiple vision tasks within a single framework, enabling more efficient and versatile learning. However, handling diverse vision tasks -- spanning dense and sparse predictions -- remains challenging due to their inherently varying output structures. In this paper, we propose AHMAD, a simple yet effective framework for generalist multitask learning that integrates different key vision tasks: semantic segmentation, instance segmentation, depth estimation, keypoint detection, and object detection. Our approach incorporates these five tasks into a unified structure: a shared encoder-decoder with several lightweight task-specific projectors. Under the multitask learning paradigm, we observed a complementary performance gain, achieving a state-of-the-art PQ of 53.1 and an mIoU of 66.5 for COCO-val panoptic and semantic segmentation, respectively. Additionally, for top-down keypoint detection, which typically incurs high computational overhead due to multiple forward passes, we introduce a knowledge distillation-based method that enables a single forward pass over the entire image, greatly improving efficiency. Ultimately, our model delivers a lightweight yet effective generalist multitask learning framework, demonstrating strong performance across five vision tasks.