Hierarchical prompt learning improves zero-shot anomaly detection and localization

Knowledge-Guided Distributional Prompt Learning with Visual Hierarchy for Zero-Shot Anomaly Detection

Computer Vision and Pattern Recognition

Summary

Detecting unusual or faulty images in new categories without prior examples is hard. The authors developed a method that learns what "normal" looks like and then identifies anomalies by comparing differences, using a special way to handle prompts that guide the model. Their approach also looks at both the whole image and specific parts to find anomalies more precisely. Tests on various industrial and medical images show it works well across different cases.

What this means in practice

  • For industrial visual inspectors: Detect unexpected defects in production parts without prior examples of faults, improving quality control accuracy.
  • For medical image analysts: Identify abnormalities in medical scans from unseen disease categories by comparing deviations from known healthy patterns.

Authors

Jiyul Ham, Seongho Yoon, Yonggon Jung, Sangjae Lee, Jun-Geol Baek

Abstract

Zero-shot anomaly detection (ZSAD) aims to detect anomalies in previously unseen classes using auxiliary images, with applications such as industrial visual inspection and medical imaging. Recent vision-language-model-based approaches improve ZSAD by aligning textual and visual representations, while distributional prompting further models prompt variability for unseen categories. However, existing methods do not explicitly combine normality-referenced distributional modeling with hierarchy-specific prompt specialization, limiting their ability to model anomalous semantics relative to normality while addressing both image-level detection and fine-grained localization. We propose Hierarchical Knowledge-Guided Distributional Prompt Learning (HiDist), a normality-referenced hierarchical distributional prompt framework for ZSAD. HiDist models each prompt token with a diagonal Gaussian distribution and generates stochastic prompt variations through Monte Carlo sampling. Rather than learning normal and anomalous distributions independently, anomalous prompt distributions are parameterized as deviations from learned normal references, explicitly encoding their dependency. The resulting distributions are specialized into global and local representations for image-level detection and pixel-level localization, respectively, while an adapted global visual representation guides hierarchical alignment. Experiments on seven industrial and medical datasets demonstrate strong cross-dataset performance across anomaly detection and localization tasks. Extensive ablation and representation analyses further support the effectiveness of the proposed distributional modeling and hierarchy-specific representation learning.