Skip to main content
SpringerLink
Log in

Spark-ITGO: a parallel invasive tumor growth optimization algorithm on spark

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The rapidly increasing complexity and scale of optimization problems pose challenges to search ability and performance of traditional evolutionary algorithms which could be only executed sequentially without scalability and are difficult to obtain an ideal solution in a reasonable time. In this paper, Spark-ITGO, a parallel and scalable invasive tumor growth optimization algorithm on Spark, is presented based on the serial invasive tumor growth optimization (ITGO). In Spark-ITGO, a parallel multiple-tumor evolution model is proposed to search the optimal solution of problems. A balanced multi-island optimal migration strategy is designed to increase diversity of population and prevent converging into a local optimum. Additionally, a universal parallel evolutionary algorithm framework is implemented based on resilient distributed dataset (RDD) and central broadcast mechanism. Spark-ITGO is evaluated on benchmark experiments of CEC2013 and CEC2010 LSGO and the results show that it achieves great scalability and performs better than other evolutionary algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig.1
Fig. 2
Fig.3
Fig. 4
Fig.5
Fig.6
Fig.7
Fig.8
Fig. 9
Fig. 10
Fig.11

Similar content being viewed by others

Data availability

The data used to support the findings of this study are available from the corresponding author upon request.

Code availability

The code of this study are available from the corresponding author upon request.

References

  1. Gong, Y.-J., Chen, W.-N., Zhan, Z.-H., Zhang, J., Li, Y., Zhang, Q., Li, J.-J.: Distributed evolutionary algorithms and their models: A survey of the state-of-the-art. Appl. Soft Comput. 34, 286–300 (2015)

    Article  Google Scholar 

  2. Teijeiro, D., Pardo, X.C., Penas, D.R., González, P., Banga, J.R., Doallo, R.: Evaluation of parallel differential evolution implementations on MapReduce and Spark. In: European Conference on Parallel Processing, pp. 397–408. Springer(2016)

  3. Alba, E., Luque, G., Nesmachnow, S.: Parallel metaheuristics: Recent advances and new trends. Int. Trans. Oper. Res. 20(1), 1–48 (2013)

    Article  Google Scholar 

  4. Ren, J., Zhang, D., He, S., Zhang, Y., Li, T.: A survey on end-edge-cloud orchestrated network computing paradigms: Transparent computing, mobile edge computing, fog computing, and cloudlet. ACM Comput. Surveys (CSUR) 52(6), 1–36 (2019)

    Article  Google Scholar 

  5. Zaharia, M., Xin, R.S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M.J.: Apache spark: A unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)

    Article  Google Scholar 

  6. Tang, D., Dong, S., Jiang, Y., Li, H., Huang, Y.: ITGO: Invasive tumor growth optimization algorithm. Appl. Soft Comput. 36, 670–698 (2015)

    Article  Google Scholar 

  7. Tang, D., Dong, S., He, L., Jiang, Y.: Intrusive tumor growth inspired optimization algorithm for data clustering. Neural Comput. Appl. 27(2), 349–374 (2016)

    Article  Google Scholar 

  8. Dubreuil, M., Gagné, C., Parizeau, M.: Analysis of a master-slave architecture for distributed evolutionary computations. IEEE Trans. Syst. Man Cybern. Part B 36(1), 229–235 (2006)

    Article  Google Scholar 

  9. Giacobini, M., Tomassini, M., Tettamanzi, A.G.B., Alba, E.: Selection intensity in cellular evolutionary algorithms for regular lattices. IEEE Trans. Evol. Comput. 9(5), 489–505 (2005)

    Article  Google Scholar 

  10. Herrera, F., Lozano, M.: Gradual distributed real-coded genetic algorithms. IEEE Trans. Evol. Comput. 4(1), 43–63 (2000)

    Article  Google Scholar 

  11. Radenski, A.: Distributed simulated annealing with MapReduce. In: European conference on the applications of evolutionary computation, pp. 466–476. Springer (2002)

  12. Ferrucci, F., Salza, P., Kechadi, M.-T., Sarro, F.: A parallel genetic algorithms framework based on Hadoop MapReduce. In: Proceedings of the 30th Annual ACM Symposium on Applied Computing, pp. 1664–1667 (2015)

  13. Lu, H.-C., Hwang, F.J., Huang, Y.-H.: Parallel and distributed architecture of genetic algorithm on Apache Hadoop and Spark. Appl. Soft Comput. 95, 106497 (2020)

    Article  Google Scholar 

  14. Ochi, L.S., Drummond, L.M., Figueiredo, R.M.: Design and implementation of a parallel genetic algorithm for the travelling purchaser problem. In: Proceedings of the 1997 ACM symposium on applied computing, pp. 257–262 (1997)

  15. Ferrucci, F., Salza, P., Sarro, F.: Using hadoop mapreduce for parallel genetic algorithms: A comparison of the global, grid and island models. Evol. Comput. 26(4), 535–567 (2018)

    Article  Google Scholar 

  16. Khezr, S.N., Navimipour, N.J.: MapReduce and its application in optimization algorithms: a comprehensive study. Majlesi J Multimed Process 4(3) (2015).

  17. Deng, C., Tan, X., Dong, X., Tan, Y.: A parallel version of differential evolution based on resilient distributed datasets model. In: Bio-inspired computing-theories and applications, pp. 84–93. Springer (2015)

  18. Teijeiro, D., Pardo, X.C., González, P., Banga, J.R., Doallo, R.: Implementing parallel differential evolution on Spark. In: european conference on the applications of evolutionary computation, pp. 75–90. Springer (2016)

  19. Qi, R.-Z., Wang, Z.-J., Li, S.-Y.J.: Technology: A parallel genetic algorithm based on spark for pairwise test suite generation. J. Comput. Sci. Technol. 31(2), 417–427 (2016)

    Article  Google Scholar 

  20. Maqbool, F., Razzaq, S., Lehmann, J., Jabeen, H.: scalable distributed genetic algorithm using apache spark (S-GA). In: International conference on intelligent computing, pp. 424–435. Springer (2019)

  21. He, Z., Peng, H., Chen, J., Deng, C., Wu, Z.: A Spark-based differential evolution with grouping topology model for large-scale global optimization. Clust. Comput. 24(1), 515–535 (2021). https://doi.org/10.1007/s10586-020-03124-z

    Article  Google Scholar 

  22. Gaifang, D., Xueliang, F., Honghui, L., Pengfei, X.: Cooperative ant colony-genetic algorithm based on spark. Comput. Electr. Eng. 60, 66–75 (2017)

    Article  Google Scholar 

  23. AlJame, M., Ahmad, I., Alfailakawi, M.: Apache Spark Implementation of Whale Optimization Algorithm. Clust. Comput. 23(3), 2021–2034 (2020)

    Article  Google Scholar 

  24. Yuan, J.: An anomaly data mining method for mass sensor networks using improved PSO algorithm based on spark parallel framework. J. Grid Comput. 18(2), 251–261 (2020)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (61976239), Science and Technology Project in Guangzhou of China (201903010046) and Innovation Foundation of High-end Scientific Research Institutions in Zhongshan of China (2019AG031).

Author information

Authors and Affiliations

  1. Guangdong Key Lab of Computer Network, School of Computer Science and Engineering, South China University of Technology, Wushan Road 381, Guangzhou, China

    Jiaqin Lin, Anghong Xiao, Shoubin Dong & Jing Zhou

Authors
  1. Jiaqin Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anghong Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shoubin Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The algorithm was conceived by AX and JL. The parallel program was developed by AX and JL. The experiments were designed and performed by SD, AX, JZ and JL. JL, AX and SD wrote, reviewed and revised the manuscript. SD guided the work. All authors read and approved the manuscript.

Corresponding author

Correspondence to Shoubin Dong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, J., Xiao, A., Dong, S. et al. Spark-ITGO: a parallel invasive tumor growth optimization algorithm on spark. Cluster Comput 25, 2633–2660 (2022). https://doi.org/10.1007/s10586-021-03396-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-021-03396-z

Keywords

Search

Navigation