Cnn-transformer improves rare attack detection on imbalanced intrusion data

Systematically Optimized CNN-Transformer with Focal Loss for Imbalanced Intrusion Detection on NSL-KDD

Cryptography and Security

Summary

Intrusion detection systems often struggle to spot rare types of attacks because the data is imbalanced, with very few examples of these attacks. The authors used a combination of convolutional neural networks and transformers, along with a special loss function called Focal Loss, to better learn from these rare attacks. They found that carefully tuning Focal Loss without oversampling techniques improved detection accuracy and balanced the results across all attack types. The study also used explanation tools to understand which features influenced their model's decisions and identified why some attack types remain challenging to detect precisely.

What this means in practice

  • For cybersecurity operations teams: Improve detection of rare network attacks in security monitoring using the optimized CNN-Transformer model with Focal Loss on classic benchmarks.
  • For machine learning engineers: Develop models for highly imbalanced classification tasks by applying tuned Focal Loss instead of oversampling to prevent precision loss.

Tested on one dataset.

Authors

Kartick Sutradhar, Ranjitha Venkatesh

Abstract

Intrusion Detection Systems (IDS) struggle with imbalanced datasets like NSL-KDD, especially in detecting rare R2L and U2R attacks. This work describes a systematically optimized and explainable framework using a CNN-Transformer architecture to improve performance on highly imbalanced data. We decided to use XGBoost for feature selection and Focal Loss as the main mechanism for minority class learning. We used Optuna for end-to-end hyrate=0.00042, batch size=256, and Focal Loss γ), using a thoughtful data splitting process to prevent data leakage. Critically, our ablation studies pointed out that while Focal Loss (optimally at γ = 1.5) substantially enhanced minority recall, adding oversampling methods like SMOTE caused precision degradation via over-correction. Our final model, using only tuned Focal Loss, achieved 98.73% overall accuracy on the NSL-KDD test data, with significantly balanced F1-scores of 84.63% (R2L) and 69.66% (U2R) over baseline approaches. Additionally, we performed SHAP analysis to understand model predictions, understand salient features (e.g., service http, logged in), and explain the persistent U2R precision challenge due to feature overlap. This study presents a comprehensive pipeline for imbalanced IDS, providing methodological insights, and demonstrates that tuned Focal Loss can be a sufficient and effective strategy for class balancing.
View PDFOpen arXiv