Abstract
Modern communications networks are complex systems, which facilitates malicious behavior. Dynamic web services are vulnerable to unknown intrusions, but traditional cyber security measures are based on fingerprinting. Anomaly detection differs from fingerprinting in that it finds events that differ from the baseline traffic. The anomaly detection methodology can be modelled with the knowledge discovery process. Knowledge discovery is a high-level term for the whole process of deriving actionable knowledge from databases. This article presents the theory behind this approach, and showcases research that has produced network log analysis tools and methods.
This article is partly based on the author’s dissertation (Sipola 2013). Author’s current affiliation is with CAP Data Technologies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brachman RJ, Anand T (1996) In: Fayyad UM, Piatetsky-Shapiro G, Smyth P, Uthurusamy R (eds) Advances in knowledge discovery and data mining, chap. The process of knowledge discovery in databases. American Association Artificial Intelligence, pp 37–57. http://dl.acm.org/citation.cfm?id=257938.257944
Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv (CSUR) 41(3):15
Craven M, Shavlik JW (1994) Using sampling and queries to extract rules from trained neural networks. In: Proceedings of the eleventh international conference on machine learning, Morgan Kaufmann, pp 37–45
Damashek M (1995) Gauging similarity with n-grams: language-independent categorization of text. Science 267(5199):843–848
David G (2009) Anomaly detection and classification via diffusion processes in hyper-networks. PhD thesis, Tel-Aviv University
David G, Averbuch A (2012) Hierarchical data organization, clustering and denoising via localized diffusion folders. Appl Comput Harmon Anal 33(1):1–23
David G, Averbuch A, Coifman R (2010) Hierarchical clustering via localized diffusion folders. In: Manifold learning and its applications: papers from the AAAI fall symposium (FS-10-06). Association for the Advancement of Artificial Intelligence (AAAI), pp 28–31
Di Pietro R, Mancini LV (eds) (2008) Intrusion detection systems. Springer, Berlin
Fayyad U, Piatetsky-Shapiro G, Smyth P (1996a) From data mining to knowledge discovery in databases. AI Maga 17(3):37–54
Fayyad U, Piatetsky-Shapiro G, Smyth P (1996b) The KDD process for extracting useful knowledge from volumes of data. Commun ACM 39(11):27–34
Fayyad UM, Piatetsky-Shapiro G, Smyth P (1996c) Knowledge discovery and data mining: towards a unifying framework. In: KDD-96 proceedings of Association for the Advancement of Artificial Intelligence (AAAI), pp 82–88
Juvonen A, Sipola T (2012) Adaptive framework for network traffic classification using dimensionality reduction and clustering. In: proceedings of the 2012 4th IEEE international congress on ultra modern telecommunications and control systems and workshops (ICUMT), New York, pp 274–279
Juvonen A, Sipola T (2013) Combining conjunctive rule extraction with diffusion maps for network intrusion detection. In: Proceedings of the 2013 IEEE symposium on computers and communications (ISCC), New York, pp 411–416
Meila M, Shi J (2001) A random walks view of spectral segmentation. In: AI and STATISTICS (AISTATS) 2001
Mukkamala S, Sung AH (2003) A comparative study of techniques for intrusion detection. In: Proceedings of the 15th IEEE international conference on tools with artificial intelligence, New York, pp 570–577
Ryman-Tubb NF, d’Avila Garcez A (2010) Soar—sparse oracle-based adaptive rule extraction: knowledge extraction from large-scale datasets to detect credit card fraud. In: Proceedings of the 2010 IEEE international joint conference on neural networks (IJCNN), New York, pp 1–9
Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888–905
Shmueli Y, Wolf G, Averbuch A (2012) Updating kernel methods in spectral decomposition by affinity perturbations. Linear Algebra Appl 437(6):1356–1365
Shmueli Y, Sipola T, Shabat G, Averbuch A (2013) Using affinity perturbations to detect web traffic anomalies. In: Proceedings of the 10th international conference on sampling theory and applications (SampTA 2013), EURASIP, Bremen, pp 444–447
Sipola T (2013) Knowledge discovery using diffusion maps. Ph.D. thesis, University of Jyväskylä
Sipola T, Juvonen A, Lehtonen J (2011) Anomaly detection from network logs using diffusion maps. In: Iliadis L, Jayne C (eds) Engineering applications of neural networks, IFIP advances in information and communication technology, vol 363. Springer, Boston, pp 172–181
Sipola T, Juvonen A, Lehtonen J (2012) Dimensionality reduction framework for detecting anomalies from network logs. Eng Intell Syst 20(1–2):87–97
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Sipola, T. (2015). Knowledge Discovery from Network Logs. In: Lehto, M., Neittaanmäki, P. (eds) Cyber Security: Analytics, Technology and Automation. Intelligent Systems, Control and Automation: Science and Engineering, vol 78. Springer, Cham. https://doi.org/10.1007/978-3-319-18302-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-18302-2_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-18301-5
Online ISBN: 978-3-319-18302-2
eBook Packages: EngineeringEngineering (R0)