Text guided image learning improves anomaly detection accuracy

TIRAD: Text-guided Image Residual Learning for Generalist Anomaly Detection

Computer Vision and Pattern Recognition

Summary

Detecting unusual or defective parts in images can be tricky, especially when the system hasn't seen those exact problems before. The authors introduce a method called TIRAD that combines both what images look like and what is described in text to spot differences between normal and defective items. By using text descriptions to focus on relevant visual details, the method better identifies anomalies. This approach works well on industrial and medical images, beating older methods that only look at pictures.

What this means in practice

  • For industrial quality control teams: Improve defect detection systems by guiding image analysis with text descriptions of anomalies to catch unseen defects more reliably.
  • For medical imaging analysts: Enhance detection of abnormalities in medical images by transferring multimodal anomaly detection techniques from industrial to medical datasets.

Authors

Seongsu Lee, Seunghyun Oh, Seungjae Moon, Youngmin Ro

Abstract

Recent anomaly detection studies have shown that residual features, which model the discrepancy between normal and abnormal patterns, provide strong generalization to anomalies unseen during training. However, existing residual-based approaches mainly rely on visual discrepancies derived from image representations, while semantic discrepancies that can be explicitly expressed through language remain largely underexplored. To address this problem, we propose TIRAD, Text-guided Image Residual Learning for Generalist Anomaly Detection, a multimodal residual learning framework that defines the discrepancy between textual representations of normal and abnormal states as a Text Residual Feature and uses it to guide the Image Residual Feature. Specifically, a Text Fusion Module converts patch-level text residuals into a gate that modulates the image residual of each patch. The gate retains the visual discrepancies that the text residual identifies as defect-related and attenuates the others, leading to more discriminative anomaly representations. Extensive experiments on industrial benchmarks and industrial-to-medical transfer demonstrate that the proposed method outperforms existing image-centric residual learning methods.