Abstract
As an important distributed real-time computation system, Storm has been widely used in a number of applications such as online machine learning, continuous computation, distributed RPC, and more. Storm is designed to process massive data streams in real time. However, there have been few studies conducted to evaluate the performance characteristics clusters in Storm. In this paper, we analyze the performance of a Storm cluster mainly from two aspects, hardware configuration and parallelism setting. Key factors that affect the throughput and latency of the Storm cluster are identified, and the performance of Storm’s fault-tolerant mechanism is evaluated, which help users use the computation system more efficiently.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Václav, S., Jana, N., Fatos, X., Leonard, B.: Geometrical and topological approaches to Big Data. Future Gener. Comput. Syst. 67, 286–296 (2017)
Chen, D.Q., et al.: Real-time or near real-time persisting daily healthcare data into HDFS and ElasticSearch Index inside a Big Data platform. IEEE Trans. Ind. Inform. 13(2), 595–606 (2017)
Mavridis, L., Karatza, H.: Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark. J. Syst. Softw. 125, 133–151 (2017)
Lv, Z.H., Song, H.B., Basanta-Val, P., Steed, A., Jo, M.: Next-generation Big Data analytics: state of the art, challenges, and future research topics. IEEE Trans. Ind. Inform. 13(4), 1891–1899 (2017)
Zhang, J., Li, C.L., Zhu, L.Y., Liu, Y.P.: The real-time scheduling strategy based on traffic and load balancing in storm. In: Proceedings of the 18th International Conference on High Performance Computing and Communications, pp. 372–379. IEEE Press (2016)
Xu, J.F., Miao, D.Q., Zhang, Y.J., Zhang, Z.F.: A three-way decisions model with probabilistic rough sets for stream computing. Int. J. Approx. Reason. 88, 1–22 (2017)
Zhang, W.S., Xu, L., Li, Z.W., Lu, Q.H., Liu, Y.: A deep-intelligence framework for online video processing. IEEE Softw. 33(2), 44–51 (2016)
Rahman, M.W., Islam, N.S., Lu, X.Y., Panda, D.K.: A comprehensive study of MapReduce over lustre for intermediate data placement and shuffle strategies on HPC clusters. IEEE Trans. Parallel Distrib. Syst. 28(3), 633–646 (2017)
Karunaratne, P., Karunasekera, S., Harwood, A.: Distributed stream clustering using micro-clusters on Apache Storm. J. Parallel Distrib. Comput. 108, 74–84 (2017)
Cardellini, V., Nardelli, M., Luzi, D.: Elastic stateful stream processing in storm. In: Proceedings of the 14th International Conference on High Performance Computing & Simulation, pp. 583–590. IEEE Press (2016)
Shieh, C.K., Huang, S.W., Sun, L.D., Tsai, M.F., Chilamkurti, N.: A topology-based scaling mechanism for Apache Storm. Int. J. Netw. Manag. 27(3), 1–12 (2017)
Li, C.L., Zhang, J., Luo, Y.L.: Real-time scheduling based on optimized topology and communication traffic in distributed real-time computation platform of storm. J. Netw. Comput. Appl. 87, 100–115 (2017)
Acknowledgment
This work is supported by the National Natural Science Foundation of China under Grant No. 61602428; the Fundamental Research Funds for the Central Universities under Grant No. 2652015338.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Yan, H., Sun, D., Gao, S., Zhou, Z. (2018). Performance Analysis of Storm in a Real-World Big Data Stream Computing Environment. In: Romdhani, I., Shu, L., Takahiro, H., Zhou, Z., Gordon, T., Zeng, D. (eds) Collaborative Computing: Networking, Applications and Worksharing. CollaborateCom 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 252. Springer, Cham. https://doi.org/10.1007/978-3-030-00916-8_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-00916-8_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00915-1
Online ISBN: 978-3-030-00916-8
eBook Packages: Computer ScienceComputer Science (R0)