Abstract
To find the strengths and weaknesses of a new multi-objective optimization algorithm, we need to compare its performance with the performances of the state-of-the-art algorithms. Such a comparison involves a selection of a performance metric, a set of benchmark problems, and a statistical test to ensure that the results are statistical significant. There are also studies in which instead of using one performance metric, a comparison is made using a set of performance metrics. All these studies assume that all involved performance metrics are equal. In this paper, we introduce a data-driven preference-based approach that is a combination of multiple criteria decision analysis with deep statistical rankings. The approach ranks the algorithms for each benchmark problem using the preference (the influence) of each performance metric that is estimated using its entropy. Experimental results show that this approach achieved similar rankings to a previously proposed method, which is based on the idea of the majority vote, where all performance metrics are assumed equal. However, as it will be shown, this approach can give different rankings because it is based not only on the idea of counting wins, but also includes information about the influence of each performance metric.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Durillo, J.J., Nebro, A.J., Alba, E.: The jMetal framework for multi-objective optimization: design and architecture. In: 2010 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2010)
Coello Coello, C.A., Lamont, G.B., Van Veldhuizen, D.A., et al.: Evolutionary Algorithms for Solving Multi-objective Problems, vol. 5. Springer, New York (2007). https://doi.org/10.1007/978-0-387-36797-2
Riquelme, N., Von Lücken, C., Baran, B.: Performance metrics in multi-objective optimization. In: Computing Conference (CLEI), 2015 Latin American, pp. 1–11. IEEE (2015)
Eftimov, T., Korošec, P., Seljak, B.K.: A novel approach to statistical comparison of meta-heuristic stochastic optimization algorithms using deep statistics. Inf. Sci. 417, 186–215 (2017)
García, S., Molina, D., Lozano, M., Herrera, F.: A study on the use of non-parametric tests for analyzing the evolutionary algorithms behaviour: a case study on the CEC2005 special session on real parameter optimization. J. Heuristics 15(6), 617–644 (2009)
Eftimov, T., Korošec, P., Koroušić Seljak, B.: Deep statistical comparison applied on quality indicators to compare multi-objective stochastic optimization algorithms. In: Nicosia, G., Pardalos, P., Giuffrida, G., Umeton, R. (eds.) MOD 2017. LNCS, vol. 10710, pp. 76–87. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-72926-8_7
Yen, G.G., He, Z.: Performance metric ensemble for multiobjective evolutionary algorithms. IEEE Trans. Evol. Compu. 18(1), 131–144 (2014)
Ravber, M., Mernik, M., Črepinšek, M.: Ranking multi-objective evolutionary algorithms using a chess rating system with quality indicator ensemble. In: 2017 IEEE Congress on Evolutionary Computation (CEC), pp. 1503–1510. IEEE (2017)
Glickman, M.E.: Example of the Glicko-2 system. Boston University (2012)
Eftimov, T., Korošec, P., Seljak, B.K.: Comparing multi-objective optimization algorithms using an ensemle of quality indicators with deep statistical comparison approach. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI) Proceedings, pp. 2801–2809. IEEE (2017)
Brans, J.P., Vincke, P.: Note - a preference ranking organisation method: (the PROMETHEE method for multiple criteria decision-making). Manag. Sci. 31(6), 647–656 (1985)
Boroushaki, S.: Entropy-based weights for multicriteria spatial decision-making. Yearb. Assoc. Pac. Coast Geogr. 79, 168–187 (2017)
Tušar, T., Filipič, B.: Differential evolution versus genetic algorithms in multiobjective optimization. In: Obayashi, S., Deb, K., Poloni, C., Hiroyasu, T., Murata, T. (eds.) EMO 2007. LNCS, vol. 4403, pp. 257–271. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-70928-2_22
Eftimov, T., Korošec, P., Seljak, B.K.: The behaviour of deep statistical comparison approach for different criteria of comparing distributions. In: Proceedings of 9th International Joint Conference on Computational Intelligence. SCITEPRESS Digital Library (2017)
Gordon, S.P.: Visualizing and understanding l’hopital’s rule. Int. J. Math. Educ. Sci. Technol. 48(7), 1096–1105 (2017)
Acknowledgments
This work was supported by the project from the Slovenian Research Agency (research core funding No. P2-0098) and from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 692286.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Eftimov, T., Korošec, P., Koroušić Seljak, B. (2018). Data-Driven Preference-Based Deep Statistical Ranking for Comparing Multi-objective Optimization Algorithms. In: Korošec, P., Melab, N., Talbi, EG. (eds) Bioinspired Optimization Methods and Their Applications. BIOMA 2018. Lecture Notes in Computer Science(), vol 10835. Springer, Cham. https://doi.org/10.1007/978-3-319-91641-5_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-91641-5_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91640-8
Online ISBN: 978-3-319-91641-5
eBook Packages: Computer ScienceComputer Science (R0)