Ensemble method improves visual anomaly detection with vision language models
Agentic Ensemble with Vision-Language Models for Visual Anomaly Detection
Artificial Intelligence
Summary
Detecting unusual or defective objects in images is hard because there are often very few examples of these anomalies to learn from. The authors designed a system called AnomalyAgent that combines the strengths of many existing detectors by smartly deciding how to mix their predictions. When detectors disagree, AnomalyAgent uses a vision-language model to choose the best way to combine their results, leading to more accurate detection. It was tested on multiple benchmarks and consistently outperformed simpler methods.
What this means in practice
- •For quality control teams: Improve automated visual inspection systems by integrating multiple anomaly detectors for more reliable detection of defects in manufacturing.
- •For medical imaging engineers: Enhance detection of abnormal findings in medical scans by combining different pretrained anomaly detection models with vision-language reasoning.
Authors
Shangquan Sun, Wenjun Luo, Xu He, Yoonsoo Lee, Shijian Lu
Abstract
Visual anomaly detection aims to detect abnormal objects, while anomaly samples are often scarce during model training. Assembling multiple pretrained detectors can help by leveraging their transferable image scores and anomaly maps. However, most existing ensemble methods apply fixed fusion rules over saved predictions which struggle when pretrained detectors clearly disagree due to different training objectives, priors, and visual representations. To this end, we design AnomalyAgent, a post-training fusion framework for accurate visual anomaly detection. AnomalyAgent aligns heterogeneous scores and maps, summarizes score- and region-level disagreement, and treats fusion as a finite action-selection problem. It implements two types of actions: 1) score actions that determine how to fuse image-level detector confidence via consensus, robust aggregation, detector weighting, or score-map consistency; 2) map actions that fuse dense anomaly responses via consensus, detector weighting, geometric aggregation, or disagreement correction. A constrained vision-language model (VLM) is introduced to inspect high-conflict samples and select valid score action and map action to produce the final prediction. We evaluate AnomalyAgent on MVTec AD and VisA as primary benchmarks, together with seven additional public anomaly benchmarks and Real-IAD for large-scale score-level evaluation. Across these benchmarks, AnomalyAgent consistently improves over equal-weight fusion and achieves the best mean image AUROC and pixel AP among the compared fusion methods on MVTec AD and VisA, while demonstrating robust transfer across additional anomaly domains.