Confidence guided method improves anomaly detection in microservice systems
Confidence-Guided Cross-Modal Knowledge Transfer for Multimodal Anomaly Detection in Microservice Systems
Software Engineering
Summary
Detecting problems in complex microservice systems is hard because data comes from different sources that can vary in reliability and meaning. The authors propose a method that measures how trustworthy each data source is and uses the reliable ones to help less reliable sources. They combine logs and performance metrics in a deep learning model that groups similar behaviors, improving problem detection. Their tests show better results than existing methods on several large datasets.
What this means in practice
- •For system reliability engineers: Identify anomalies in production microservice setups more accurately by using confidence-guided fusion of logs and metrics.
- •For cloud operations teams: Improve monitoring tools by integrating cross-modal knowledge transfer to handle variable data quality and heterogeneous sources.
Authors
Peipeng Wang, Xiuguo Zhang, Lihua Zhang, Zhiying Cao, Jingchun Zhou, Zheng Li
Abstract
Accurate anomaly detection is essential for reliable and secure operations of microservice systems. While an increasing number of studies have shifted from unimodal modeling to multimodal interaction and fusion, effectively leveraging reliable cross-modal information remains challenging. The challenge primarily stems from two aspects. Firstly, different modalities are influenced by factors like load fluctuations, leading to dynamically changing reliability. Secondly, multimodal data exhibit heterogeneity in both structure and semantics. Therefore, we propose a confidence-guided Cross-Modal knowledge Transfer method for multimodal Anomaly Detection (CMT-AD). It jointly models metrics and logs within a unified deep clustering framework and estimates modality reliability through the soft clustering distributions, where clustering uncertainty is quantified into confidence scores. Guided by these confidence scores, the model actively analyzes the contributions of each modality in cross-modal interactions and supplements low-confidence modalities with knowledge from high-confidence ones. To further mitigate cross-modal heterogeneity, we introduce a gated intermediate modality and design structural and semantic consistency constraints that align the original modalities with the intermediate modality to preserve similarity structures and semantic distributions across modalities. Furthermore, intra-modal and cross-modal regularization terms are incorporated to enhance cluster compactness and mitigate negative transfer. We evaluated CMT-AD on three large-scale datasets, and the results demonstrate that CMT-AD outperforms state-of-the-art approaches and achieves an F1-score higher than 0.9.