Abstract
Logging and monitoring data is very important during the development process as well as the operation of information systems. As the data grows to TB every day, this problem becomes more complicated. Companies can generally buy big data analytics platforms or build it by themselves. Whether buying or building, it is important to have a realistic expectation of time and budget needed to successfully implement, roll out and provide ongoing support. There was a lot of confusion and frustration as the data platform market grew. Suppliers sell their capabilities instead of the actual needs of their customers. Contrary to that trend, some companies would like to build the platform using open source systems such as Apache Flume, Apache Spark Streaming and some other auxiliary technologies at a reasonable cost. This study analyzes requirements, introduces system architecture, and builds a logging and monitoring system for streaming data. The work is also a real project in the field of advertising.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Deha solution. https://deha.co.jp/
Srinivasa, K.G., Siddesh, G.M., Srinidhi, H.: Apache Flume, Network Data Analytics, pp 95–107 (2018). https://doi.org/10.1007/978-3-319-77800-6
Tenesaca-Luna, G.A., Imba D., Mora-Arciniegas, M.B., Segarra-Faggioni, V., RamÃrez-Coronel, R.L.: Use of apache flume in the big data environment for processing and evaluation of the data quality of the twitter social network. In: Botto-Tobar, M., Barba-Maggi, L., González-Huerta, J., Villacrés-Cevallos, P., Gómez, O.S., Uvidia-Fassler, M. (eds) Information and Communication Technologies of Ecuador (TIC.EC). TICEC 2018. Advances in Intelligent Systems and Computing, vol. 884. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-02828-2_23
Nasiri, H., Nasehi, S., Goudarzi, M.: Evaluation of distributed stream processing frameworks for IoT applications in Smart Cities. J. Big Data 6(1), 1–24 (2019). https://doi.org/10.1186/s40537-019-0215-2
Sharma, S.: Dynamic hashtag interactions and recommendations: an implementation using apache spark streaming and GraphX. In: Sharma, N., Chakrabarti, A., Balas, V.E. (eds.) Data Management, Analytics and Innovation. AISC, vol. 1042, pp. 723–738. Springer, Singapore (2020). https://doi.org/10.1007/978-981-32-9949-8_51
Spark Streaming Programming Guide (2020). https://spark.apache.org/docs/latest/streaming-programming-guide.html
(2020). https://flume.apache.org/
(2020). https://www.elastic.co/kibana
(2020). https://grafana.com/grafana/
(2020). https://laravel.com/docs/7.x
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chung, N.N., Hung, P.D. (2020). Logging and Monitoring System for Streaming Data. In: Luo, Y. (eds) Cooperative Design, Visualization, and Engineering. CDVE 2020. Lecture Notes in Computer Science(), vol 12341. Springer, Cham. https://doi.org/10.1007/978-3-030-60816-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-60816-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60815-6
Online ISBN: 978-3-030-60816-3
eBook Packages: Computer ScienceComputer Science (R0)