Skip to main content
SpringerLink
Log in

A scalable network intrusion detection system towards detecting, discovering, and learning unknown attacks

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Network intrusion detection systems (IDSs) based on deep learning have reached fairly accurate attack detection rates. But these deep learning approaches usually have been performed in a closed-set protocol that only known classes appear in training are considered during classification, the existing IDSs will fail to detect the unknown attacks and misclassify them as the training known classes, hence are not scalable. Furthermore, these IDSs are not efficient for updating the deep detection model once new attacks are discovered. To address those problems, we propose a scalable IDS towards detecting, discovering, and learning unknown attacks, it has three components. Firstly, we propose the open-set classification network (OCN) to detect unknown attacks, OCN based on the convolutional neural network adopts the nearest class mean (NCM) classifier, two new loss are designed to jointly optimize it, including Fisher loss and maximum mean discrepancy (MMD) loss. Subsequently, the semantic embedding clustering method is proposed to discover the hidden unknown attacks from all unknown instances detected by OCN. Then we propose the incremental nearest cluster centroid (INCC) method for learning the discovered unknown attacks through updating the NCM classifier. Extensive experiments on KDDCUP’99 dataset and CICIDS2017 dataset indicate that our OCN outperforms the state-of-the-art comparison methods in detecting multiple types of unknown attacks. Our experiments also verify the feasibility of the semantic embedding clustering method and INCC in discovering and learning unknown attacks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://github.com/zhangzhao156/scalable-NIDS.

References

  1. Karatas G, Demir O, Sahingoz OK (2018) Deep learning in intrusion detection systems. In: 2018 International congress on big data, deep learning and fighting cyber terrorism (IBIGDELFT), IEEE, pp 113–116

  2. Buczak AL, Guven E (2016) A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Commun Surv Tutor 18(2):1153–1176

    Article  Google Scholar 

  3. Rudd EM, Rozsa A, Günther M et al (2017) A survey of stealth malware: attacks, mitigation measures, and steps toward autonomous open world solutions. IEEE Commun Surv Tutor 19(2):1145–1172

    Article  Google Scholar 

  4. Scheirer WJ, de Rezende Rocha A, Sapkota A, Boult TE (2013) Toward open set recognition. IEEE Trans Pattern Anal Mach Intell 35(7):1757–1772

    Article  Google Scholar 

  5. Bendale A, Boult T (2015) Towards open world recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1893–1902

  6. Miller D, Sünderhauf N, Milford M et al (2020) Class anchor clustering: a distance-based loss for training open set classifiers. arXiv preprint arXiv:2004.02434

  7. Geng C, Huang S, Chen S (2020) Recent advances in open set recognition: a survey. IEEE Trans Pattern Anal Mach Intell, early access. https://doi.org/10.1109/TPAMI.2020.2981604

  8. Rudd EM, Jain LP, Scheirer WJ et al (2017) The extreme value machine. IEEE Trans Pattern Anal Mach Intell 40(3):762–768

    Article  Google Scholar 

  9. Jain LP, Scheirer WJ, Boult TE (2014) Multi-class open set recognition using probability of inclusion. In: European conference on computer vision, Springer, Cham, pp 393–409

  10. Henrydoss J, Cruz S, Rudd EM et al (2017) Incremental open set intrusion recognition using extreme value machine. In: 2017 16th IEEE international conference on machine learning and applications (ICMLA), IEEE, pp 1089–1093

  11. Cruz S, Coleman C, Rudd EM et al (2017) Open set intrusion recognition for fine-grained attack categorization. In: 2017 IEEE international symposium on technologies for homeland security (HST), IEEE, pp 1–6

  12. Hubballi N, Suryanarayanan V (2014) False alarm minimization techniques in signature-based intrusion detection systems: a survey. Comput Commun 49:1–17

    Article  Google Scholar 

  13. Agarwal M, Pasumarthi D, Biswas S et al (2016) Machine learning approach for detection of flooding DoS attacks in 802.11 networks and attacker localization. Int J Mach Learn Cybern 7(6):1035–1051

    Article  Google Scholar 

  14. Ashfaq RAR, He Y, Chen D (2017) Toward an efficient fuzziness based instance selection methodology for intrusion detection system. Int J Mach Learn Cybern 8(6):1767–1776

    Article  Google Scholar 

  15. Yan Q, Wang M, Huang W et al (2019) Automatically synthesizing DoS attack traces using generative adversarial networks. Int J Mach Learn Cybern 10(12):3387–3396

    Article  Google Scholar 

  16. Roopak M, Tian GY, Chambers J (2019) Deep learning models for cyber security in IoT networks. In: 2019 IEEE 9th annual computing and communication workshop and conference (CCWC), IEEE, pp 0452–0457

  17. Zhang Y, Chen X, Jin L et al (2019) Network intrusion detection: based on deep hierarchical network and original flow data. IEEE Access 7:37004–37016

    Article  Google Scholar 

  18. Khan MA, Karim M, Kim Y (2019) A scalable and hybrid intrusion detection system based on the convolutional-LSTM network. Symmetry 11(4):583

    Article  Google Scholar 

  19. Lin P, Ye K, Xu CZ (2019) Dynamic network anomaly detection system by using deep learning techniques. In: International conference on cloud computing, Springer, Cham, pp 161–176

  20. Hendrycks D, Gimpel K (2016) A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136

  21. Liang S, Li Y, Srikant R (2017) Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690

  22. Shu L, Xu H, Liu B (2018) Unseen class discovery in open-world classification. arXiv preprint arXiv:1801.05609

  23. Hsu YC, Lv Z, Schlosser J et al (2018) A probabilistic constrained clustering for transfer learning and image category discovery. arXiv preprint arXiv:1806.11078

  24. Shmelkov K, Schmid C, Alahari K (2017) Incremental learning of object detectors without catastrophic forgetting. In: Proceedings of the IEEE international conference on computer vision, pp 3400–3409

  25. Rebuffi SA, Kolesnikov A, Sperl G et al (2017) icarl: Incremental classifier and representation learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2001–2010

  26. Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572

  27. Sriperumbudur Bharath K, Kenji F, Arthur G, Lanckriet Gert RG, Scholkopf B (2009) Kernel choice and classifiability for RKHS embeddings of probability distributions. Adv Neural Inf Process Syst 22:1750–1758

    Google Scholar 

  28. Long M, Wang J (2015) Learning transferable features with deep adaptation networks. In: Proceedings of the 32nd international conference on machine learning (ICML), pp 97–105

  29. Changpinyo S, Chao WL, Sha F (2017) Predicting visual exemplars of unseen classes for zero-shot learning. In: Proceedings of the IEEE international conference on computer vision, pp 3476–3485

  30. Ester M, Kriegel, Hans-Peter, Sander J et al (1996) A density-based algorithm for discovering clusters a density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the 2nd international conference on knowledge discovery & data mining (KDD'96), pp 226–231

  31. Schubert E, Sander Jörg, Ester M et al (2017) DBSCAN revisited, revisited: why and how you should (still) use DBSCAN. ACM Trans Database Syst 42(3):1–21

    Article  MathSciNet  Google Scholar 

  32. Mensink T, Verbeek J, Perronnin F et al (2013) Distance-based image classification: generalizing to new classes at near-zero cost. IEEE Trans Pattern Anal Mach Intell 35(11):2624–2637

    Article  Google Scholar 

  33. Zhang Y, Chen X, Guo D et al (2019) PCCN: parallel cross convolutional neural network for abnormal network traffic flows detection in multi-class imbalanced network traffic flows. IEEE Access 7:119904–119916

    Article  Google Scholar 

  34. Long M, Zhu H, Wang J et al (2016) Deep transfer learning with joint adaptation networks. arXiv preprint arXiv:1605.06636

  35. Yang Y, Xu D, Nie F et al (2010) Image clustering using local discriminant models and global integration. IEEE Trans Image Process 19(10):2761–2773

    Article  MathSciNet  Google Scholar 

  36. Kuhn HW (1955) The Hungarian method for the assignment problem. Nav Res Logist Q 2(1–2):83–97

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work is supported by National Key R&D Program of China under Grant No.2020YFC1522503.

Author information

Authors and Affiliations

  1. School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Zhao Zhang, Yong Zhang, Da Guo & Mei Song

  2. Beijing Key Laboratory of Work Safety Intelligent Monitoring, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Yong Zhang

Authors
  1. Zhao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Da Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Zhang, Y., Guo, D. et al. A scalable network intrusion detection system towards detecting, discovering, and learning unknown attacks. Int. J. Mach. Learn. & Cyber. 12, 1649–1665 (2021). https://doi.org/10.1007/s13042-020-01264-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01264-7

Keywords

Search

Navigation