Abstract
Log files usually contain very rich running information of the software system, which can be used for anomaly detection, performance modeling, and failure diagnosis, etc. In a large-scale deployment system, log records are always unstructured and can not directly use for log analysis. Log parsing, as a key prerequisite for log analysis, converts unstructured log records into structured event templates by extracting the constant portion of the raw log. Traditionally, log parsing can be achieved by manually using the regular expression, which requires many experts knowledge and has very low efficiency. Therefore, the accuracy and efficiency of log parsing are very important, especially in large-scale distributed systems. In this paper, we propose an efficient algorithm namely CLF (Clustering based on Length and First token) for extracting log event templates from raw log based on heuristic rules. The CLF algorithm works through a 3-step process: clustering unstructured logs based on heuristic rules, clustering again according to specific separation rules and finally generating event templates. Finally, we used 7 data sets to evaluate the performance of CLF and compared with three state-of-the-art log parser algorithms, where CLF ranks higher on most of the data sets and also has advantages in execution time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bertero, C., Roy, M., Sauvanaud, C., Tredan, G.: Experience report: log mining using natural language processing and application to anomaly detection. In: IEEE International Symposium on Software Reliability Engineering (2017)
Goldstein, M., Raz, D., Segall, I.: Log-based behavioral differencing. In: IEEE International Symposium on Software Reliability Engineering, Experience report (2017)
Kc, K., Gu, X.: ELT: Efficient log-based troubleshooting system for cloud computing infrastructures. In: Reliable Distributed Systems (2011)
Ren, R., Fu, X., Zhan, J., Zhou, W.: LogMaster: Mining event correlations in logs of large scale cluster systems. In: IEEE Symposium on Reliable Distributed Systems (2012)
Zou, D.Q.: Uilog: improving log-based fault diagnosis by log analysis. J. Comput. Sci. Technol. 31(5), 1038–1052 (2016)
Wong, W.E., Debroy, V., Golden, R., Xu, X., Thuraisingham, B.: Effective software fault localization using an RBF neural network. IEEE Transact. Reliab. 61(1), 149–169 (2012)
Reidemeister, T., Jiang, M., Ward, P.A.S.: Mining unstructured log files for recurrent fault diagnosis. In: IFIP/IEEE International Symposium on Integrated Network Management (2011)
Nagaraj, K., Neville, J., Killian, C.: Structured comparative analysis of systems logs to diagnose performance problems. In: USENIX Conference on Networked Systems Design and Implementation (2012)
Fu, Q., Lou, J.G., Wang, Y., Li, J.: Execution anomaly detection in distributed systems through unstructured log analysis. In: IEEE International Conference on Data Mining (2009)
Lou, J.G., Fu, Q., Yang, S., Xu, Y., Li, J.: Mining invariants from console logs for system problem detection. In: Proceedings of USENIX ATC, pp. 231–244 (2010)
Wei, X., Ling, H., Fox, A., Patterson, D.A., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: ACM SIGOPS Symposium on Operating Systems Principles (2009)
Shima, K.: Length matters: clustering system log messages using length of words (2016)
Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., Sasnauskas, R.: A search-based approach for accurate identification of log message formats. pp. 167–177 (2018)
Ma, H., Hellerstein, J.L.: Mining partially periodic event patterns with unknown periods. In: International Conference on Data Engineering (2000)
Zheng, Q., Xu, K., Lv, W., Ma, S.: Intelligent search of correlated alarms from database containing noise data. In: Network Operations and Management Symposium (2001)
Stearley, J.: Towards informatic analysis of syslogs. In: IEEE International Conference on Cluster Computing (2004)
Vaarandi, R.: A data clustering algorithm for mining patterns from event logs. In: IP Operations and Management (2003)
Liang, T., Tao, L., Perng, C.S.: LogSig: generating system events from raw textual logs. In: ACM International Conference on Information and Knowledge Management (2011)
Mizutani, M.: Incremental mining of system log format. In: IEEE International Conference on Services Computing (2013)
Hamooni, H., Debnath, B., Xu, J., Hui, Z., Mueen, A.: Logmine: Fast pattern recognition for log analytics. In: ACM International on Conference on Information and Knowledge Management (2016)
Nagappan, M., Vouk, M.A.: Abstracting log lines to log event types for mining software system logs. In: Mining Software Repositories (2010)
Vaarandi, R., Pihelgas, M.: LogCluster - a data clustering and pattern mining algorithm for event logs. In: International Conference on Network and Service Management (2016)
Jiang, Z.M., Hassan, A.E., Flora, P., Hamann, G.: Abstracting execution logs to execution events for enterprise applications (short paper). pp. 181–186, August 2008
Makanju, A., Zincir-Heywood, A.N., Milios, E.E.: A lightweight algorithm for message type extraction in system application logs. IEEE Transact. Knowl. Data Eng. 24(11), 1921–1936 (2012)
He, P., Zhu, J., Zheng, Z., Lyu, M.R.: Drain: an online log parsing approach with fixed depth tree. In: IEEE International Conference on Web Services (2017)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transact. Evol. Comput. 6(2), 182–197 (2002)
Makanju, A., Zincir-Heywood, A.N., Milios, E.E.: Clustering event logs using iterative partitioning. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2009)
Liang, T., Tao, L.: LogTree: a framework for generating system events from raw textual logs. In: IEEE International Conference on Data Mining (2011)
Zhu, J., He, S., Liu, J., He, P., Lyu, M.R.: Tools and benchmarks for automated log parsing (2018)
Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval (2010)
Acknowledgment
This work is partially supported by the National Key Research and Development Program of China (2016YFC0400709), the Next Generation Internet Technology Innovation Project of CERNET (NGII20180306), the Science and Technology Commission of Tianjin Binhai New Area (BHXQKJXM-PT-ZJSHJ-2017005), the Natural Science Foundation of Tianjin (18YFYZCG00060) and Nankai University (91922299).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, L., Xie, X., Xie, K., Wang, Z., Lu, Y., Zhang, Y. (2019). An Efficient Log Parsing Algorithm Based on Heuristic Rules. In: Yew, PC., Stenström, P., Wu, J., Gong, X., Li, T. (eds) Advanced Parallel Processing Technologies. APPT 2019. Lecture Notes in Computer Science(), vol 11719. Springer, Cham. https://doi.org/10.1007/978-3-030-29611-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-29611-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29610-0
Online ISBN: 978-3-030-29611-7
eBook Packages: Computer ScienceComputer Science (R0)
