Skip to main content
SpringerLink
Log in

AEGR: a simple approach to gradient reversal in autoencoders for network anomaly detection

  • Application of soft computing
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Anomaly detection is referred to a process in which the aim is to detect data points that follow a different pattern from the majority of data points. With the rapid development of computer technology, protecting networks from various threats such as network intruders is becoming crucial. Traditional anomaly detection methods suffer from several well-known challenges that hinder their performance, such as the curse of high dimensionality. Autoencoders are unsupervised neural networks that have been used for the purpose of reducing dimensionality and also detecting network anomalies in large datasets. The performance of autoencoders debilitates when the training set contains noise and anomalies. In this paper, a new gradient-reversal method is proposed to overcome the influence of anomalies on the training phase for the purpose of detecting network anomalies. The method is different from other approaches as it does not require an anomaly-free training set and is based on reconstruction error. Once latent variables are extracted from the network, local outlier factor is used to separate normal data points from anomalies. A simple pruning approach and data augmentation are also added to improve performance further. The experimental results show that the proposed model can outperform other well-known approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Arzoo M, Neeraj K (2019) Deep learning models for traffic flow prediction in autonomous vehicles: a review, solutions, and challenges. Vehicular Commun 20:100184

    Article  Google Scholar 

  • Aygun RC, Yavuz AG (2017) Network anomaly detection with stochastically improved autoencoder based models. In: 2017 IEEE 4th international conference on cyber security and cloud computing (CSCloud), pp 193–198. IEEE

  • Bottou L (1991) Stochastic gradient learning in neural networks. Proc Neuro-Nımes 91(8):12

    Google Scholar 

  • Breunig MM, Kriegel HP, Ng RT, Sander J (2000) LOF: identifying density-based local outliers. SIGMOD Rec., 29(2):93–104, may

  • Breuniq MM, Kriegel HP, Ng RT, Sander J (2000) LOF: identifying density-based local outliers. SIGMOD Record (ACM Special Interest Group on Management of Data) 29(2):93–104

    Google Scholar 

  • Cao VL, Nicolau M, McDermott J (2019) Learning neural representations for network anomaly detection. IEEE Trans Cybernet 49(8):3074–3087

    Article  Google Scholar 

  • Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3):15:1–15:58

    Article  Google Scholar 

  • Charte D, Charte F, García S, del Jesus MJ, Herrera F (2018) A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines. Information Fusion 44:78–96

    Article  Google Scholar 

  • Chen J, Sathe S, Aggarwal C, Turaga D (2017) Outlier detection with autoencoder ensembles. In: Proceedings of the 2017 SIAM international conference on data mining, pp 90–98. SIAM

  • Chen Z, Yeo CK, Lee BS, Lau CT(2018) Autoencoder-based network anomaly detection. In: 2018 wireless telecommunications symposium (WTS), pp. 1–5

  • Chow JK, Su Z, Wu J, Tan PS, Mao X, Wang YH (2020) Anomaly detection of defects on concrete structures with the convolutional autoencoder. Adv Eng Inf 45:101105

    Article  Google Scholar 

  • Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7(Jan):1–30

    MathSciNet  MATH  Google Scholar 

  • Dheeru D, Taniskidou EK (2017) UCI machine learning repository

  • Erfani SM, Rajasegarar S, Karunasekera S, Leckie C (2016) High-dimensional and large-scale anomaly detection using a linear one-class svm with deep learning. Pattern Recogn 58:121–134

    Article  Google Scholar 

  • Ganin Y, Lempitsky V (2014) Unsupervised domain adaptation by backpropagation. arXiv:1409.7495

  • García S, Grill M, Stiborek J, Zunino A (2014) An empirical comparison of botnet detection methods. Comput Secur 45:100–123

    Article  Google Scholar 

  • Garg S, Kaur K, Kumar N, Rodrigues JJPC (2019) Hybrid deep-learning-based anomaly detection scheme for suspicious flow detection in SDN: a social multimedia perspective. IEEE Trans Multimedia 21(3):566–578

    Article  Google Scholar 

  • Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  • Hodge Victoria J, Jim A (2004) A survey of outlier detection methodologies. Artif Intell Rev 22(2):85–126

    Article  Google Scholar 

  • Kerby DS (2014) The simple difference formula: an approach to teaching nonparametric correlation. Comprehens Psychol 3:11–IT

    Article  Google Scholar 

  • Khan Shehroz S, Madden Michael G (2010) A survey of recent trends in one class classification. In: Coyle L, Freyne J (eds) Artificial intelligence and cognitive science: 20th irish conference, AICS 2009, Dublin, Ireland, August 19–21, 2009, Revised Selected Papers. Springer, Berlin, pp 188–197

    Chapter  Google Scholar 

  • Lian J, Jia W, Zareapoor M, Zheng Y, Luo R, Jain DK, Kumar N (2020) Deep-learning-based small surface defect detection via an exaggerated local variation-based generative adversarial network. IEEE Trans Ind Inf 16(2):1343–1351

    Article  Google Scholar 

  • Ma Y, Zhang P, Cao Y, Guo L (2013) Parallel auto-encoder for efficient outlier detection. In: Big Data, 2013 IEEE international conference on, pp 15–17. IEEE

  • Moustafa N, Slay J (2015) UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In: 2015 Military communications and information systems conference (MilCIS), pp 1–6, nov

  • Munir M, Siddiqui SA, Dengel A, Ahmed S (2019) DeepAnT: a deep learning approach for unsupervised anomaly detection in time series. IEEE Access 7:1991–2005

    Article  Google Scholar 

  • Nguyen HD, Tran KP, Thomassey S, Hamad M (2021) Forecasting and anomaly detection approaches using lstm and lstm autoencoder techniques with the applications in supply chain management. Int J Inf Manag 57:102282

    Article  Google Scholar 

  • Provost F, Fawcett T (1997) Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In: Proceedings of the third international conference on knowledge discovery and data mining, KDD’97, pp 43–48. AAAI Press

  • Qi Y, Wang Y, Zheng X, Wu Z (2014) Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 6716–6720. IEEE

  • Safara F, Souri A, Serrizadeh M (2020) Improved intrusion detection method for communication networks using association rule mining and artificial neural networks. IET Commun 14(7):1192–1197

    Article  Google Scholar 

  • Sakurada M, Yairi T (2014) Anomaly detection using autoencoders with nonlinear dimensionality reduction. In: Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis, p 4. ACM

  • Schreyer M, Sattarov T, Borth D, Dengel A, Reimer B (2017) Detection of anomalies in large scale accounting data using deep autoencoder networks. arXiv:1709.05254

  • Sun Z, Sun H (2019) Stacked denoising autoencoder with density-grid based clustering method for detecting outlier of wind turbine components. IEEE Access 7:13078–13091

  • Sun J, Wang X, Xiong N, Shao J (2018) Learning sparse representation with variational auto-encoder for anomaly detection. IEEE Access 6:33353–33361

    Article  Google Scholar 

  • Su S, Xiao L, Zhang Z, Gu F, Ruan L, Li S, He Z, Huo Z, Yan B, Wang H, et al (2017) N2DLOF: a new local density-based outlier detection approach for scattered data. In: High performance computing and communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 2017 IEEE 19th International Conference on, pp 458–465. IEEE

  • Tang B, He H (2017) A local density-based approach for outlier detection. Neurocomputing 241:171–180

    Article  Google Scholar 

  • Tavallaee M, Bagheri E, Lu W, Ghorbani AA (2009) A detailed analysis of the KDD CUP 99 data set. In: 2009 IEEE symposium on computational intelligence for security and defense applications, pp 1–6. IEEE

  • Van Cao L, Nicolau M, McDermott J (2016) A hybrid autoencoder and density estimation model for anomaly detection. In: Handl J, Hart E, Lewis PR, López-Ibáñez M, Ochoa G, Paechter B (eds) Parallel problem solving from nature–PPSN XIV. Springer, Cham, pp 717–726

    Chapter  Google Scholar 

  • Wang Y, Yao H, Zhao S (2016) Auto-encoder based dimensionality reduction. Neurocomputing 184:232–242

    Article  Google Scholar 

  • Yousefi-Azar M, Varadharajan V, Hamey L, Tupakula U (2017) Autoencoder-based feature learning for cyber security applications. In: 2017 International joint conference on neural networks (IJCNN), pp 3854–3861. IEEE

  • Zhang Q, Yang LT, Chen Z, Li P (2018) A survey on deep learning for big data. Inf Fusion 42:146–157

    Article  Google Scholar 

  • Zhou C, Paffenroth RC (2017) Anomaly detection with robust deep autoencoders. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 665–674. ACM

Download references

Funding

The authors acknowledge that this study has not been funded.

Author information

Authors and Affiliations

  1. University of Nottingham Malaysia, Semenyih, Malaysia

    Kasra Babaei, Zhi Yuan Chen & Tomas Maul

Authors
  1. Kasra Babaei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi Yuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tomas Maul
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhi Yuan Chen.

Ethics declarations

Conflict of interest

All authors have participated in every stage of the research including but not limited to experimenting, analysis, drafting, reviewing and revising the work. All authors declare that they have no conflicts of interest.

Research involving human participants and/or animals

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Babaei, K., Chen, Z.Y. & Maul, T. AEGR: a simple approach to gradient reversal in autoencoders for network anomaly detection. Soft Comput 25, 15269–15280 (2021). https://doi.org/10.1007/s00500-021-06110-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-021-06110-8

Keywords

Search

Navigation