Skip to main content
SpringerLink
Log in

Cost-Performance Modeling with Automated Benchmarking on Elastic Computing Clouds

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

The importance of evaluating performance of cloud systems has been increasing with the rapid growing market demands for cloud computing. However, the performance testers often have to go through the hassle of tedious manual operations when interacting with the cloud. A cloud performance evaluation framework is designed for both broad cloud support and good workload extensibility, which provides an automatic interface to monitor the capability and scalability of Infrastructure-as-a-Service cloud systems. Cloud API modules are implemented for Amazon EC2 service and OpenStack. It can achieve flexible control workflows for multiple of different workloads and user customization to test scenarios. With several built-in workloads and metric aggregation methods, a series of tests is performed on our private clouds to compare the performance and scalability from multiple aspects. A methodology is also proposed to build a cost-performance model to better understand and analyze the efficiency of different types of cloud systems. Based on the results of the experiments, the model indicates a polynomial relation between performance per instance and the overall cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bhaskar Prasad, R., Admela, J., Dimitrios, K., Yves, G.: Architectural Requirements for Cloud Computing Systems: An Enterprise Cloud Approach. J. Grid Comput. 9(1), 3–26 (2011)

    Article  Google Scholar 

  2. Thomas, R., Geoff, C., et al.: Grid and cloud computing: opportunities for integration with the next generation network. J. Grid Comput. 7(3), 375–393 (2009)

    Article  Google Scholar 

  3. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., et al.: A view of cloud computing. Commun. ACM 53(4), 50–58 (2010)

    Article  Google Scholar 

  4. Mell, P., Grance, T.: The nist definition of cloud computing. NIST Spec. Publ. 800(145), 7 (2011)

    Google Scholar 

  5. Wang, L., Von Laszewski, G., Younge, A., He, X., Kunze, M., Tao, J., Fu, C.: Cloud computing: a perspective study. New Gener. Comput. 28(2), 137–146 (2010)

    Article  MATH  Google Scholar 

  6. Costa, P., Migliavacca, M., Pietzuch, P., Wolf, A.L.: Naas: network-as-a-service in the cloud. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services, Hot-ICE, vol. 12, pp. 1-1. USENIX, Berkeley (2012)

  7. Shiraz, M., Gani, A., Azra, S., Khan, S., Ahmad, R.W.: Energy efficient computational offloading framework for mobile cloud computing. J. Grid Comput. 13(1), 1–18 (2015)

    Article  Google Scholar 

  8. Yangui, S., Marshall, I.J., Laisne, J.P., Tata, S.: CompatibleOne: the open source cloud broker. J. Grid Comput. 12(1), 93–109 (2014)

    Article  Google Scholar 

  9. van Vliet, J., Paganelli, F.: Programming Amazon EC2. O’Reilly Media, Sebastopol (2011)

    Google Scholar 

  10. Krishnan, S.: Programming Microsoft Azure. O’Reilly Media, Sebastopol (2010)

    Google Scholar 

  11. Hp helion public cloud. http://www.hpcloud.com/

  12. Sabharwal, N., Shankar, R.: Apache CloudStack Cloud Computing. Packt Publishing, Birmingham (2013)

    Google Scholar 

  13. Toraldo, G.: OpenNebula3 Cloud Computing. Packt Publishing, Birmingham (2012)

    Google Scholar 

  14. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: 2009 9th IEEE International Symposium on Cluster Computing and the Grid, CCGRID’09, IEEE, pp. 124–131. IEEE, Piscataway (2009)

  15. Jackson, K., Bunch, C.: OpenStack Cloud Computing Cookbook, 2nd edn. Packt Publishing, Birmingham (2012)

    Google Scholar 

  16. Avetisyan, A.I., Campbell, R., Gupta, I., Heath, M.T., Ko, S.Y., Ganger, G.R., Kozuch, M.A., O’Hallaron, D., Kunze, M., Kwan, T.T., et al.: Open cirrus: a global cloud computing testbed. Computer 43(4), 35–43 (2010)

    Article  Google Scholar 

  17. Keqin, L.: Optimal load distribution for multiple heterogeneous blade servers in a cloud computing environment. J. Grid Comput. 11(1), 27–46 (2013)

    Article  Google Scholar 

  18. Ge, X., Qi, Z., Chen, K., Duan, J., Dong, Z.: Loosely-coupled benchmark framework automates performance modeling on iaas clouds. In: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, UCC 2014. IEEE, Piscataway (2014)

  19. Casazza, J.P., Greenfield, M., Shi, K.: Redefining server performance characterization for virtualization benchmarking. Intel Technol. J. 3, 10 (2006)

    Google Scholar 

  20. Makhija, V., Herndon, B., Smith, P., Roderick, L., Zamost, E., Anderson, J.: Vmmark: a scalable benchmark for virtualized systems. VMware Inc, CA, Tech. Rep. VMware-TR-2006-002 (2006)

  21. Muñoz, V.M., Ramo, A.C., Diaz, R.G., Tsaregorodtsev, A.: Cloud governance by a credit model with DIRAC. In: Proceedings of the 4th International Conference on Cloud Computing and Services Science, pp. 679–686 (2014)

  22. Zheng, X., Martin, P., Brohman, K., Xu, L.D.: Cloud service negotiation in internet of things environment: A mixed approach. IEEE Trans. Ind. Inf. 10(2), 1506–1515 (2014)

    Article  Google Scholar 

  23. Tao, F., Laili, Y., Xu, L., Zhang, L.: Fc-paco-rm: A parallel method for service composition optimal-selection in cloud manufacturing system. IEEE Trans. Ind. Inf. 9(4), 2023–2033 (2013)

    Article  Google Scholar 

  24. Wang, L., Zhan, J., Luo, C., Zhu, Y., Yang, Q., He, Y., Gao, W., Jia, Z., Shi, Y., Zhang, S., Zheng, C., Lu, G., Zhan, K., Li, X., Qiu, B.: Bigdatabench: A big data benchmark suite from internet services. In: Proceedings of the 20th IEEE International Symposium on High Performance Computer Architecture, HPCA’2014. Piscataway, NJ, USA: IEEE, pp. 488–499 (2014)

  25. Zheng, X., Martin, P., Brohman, K., Xu, L.D.: Cloudqual: A quality model for cloud services. IEEE Trans. Ind. Inf. 10(2), 1527–1536 (2014)

    Article  Google Scholar 

  26. Tao, F., Zuo, Y., Xu, L.D., Zhang, L.: Iot-based intelligent perception and access of manufacturing resource toward cloud manufacturing. IEEE Trans. Ind. Inf. 10(2), 1547–1557 (2014)

    Article  Google Scholar 

  27. Muñoz, V.M., Ramo, A.C., Albor, V.F., Diaz, R.G., Arévalo, G.M.: Rafhyc: an architecture for constructing resilient services on federated hybrid clouds. J. Grid Comput. 11(4), 753–770 (2013)

    Article  Google Scholar 

  28. Cloudharmony: simplify the comparison of cloud services, Laguna Beach, CA, USA, http://cloudharmony.com/

  29. Li, A., Yang, X., Kandula, S., Zhang, M.: Cloudcmp: comparingpublic cloud providers. In: Proceedings of the 2010 conferenceon Internet measurement conference, ACM. NewYork, NY, USA: ACM, pp. 1–14 (2010)

  30. Novakovic, D., Vasic, N., Novakovic, S., Kostic, D., Bianchini, R.: Deepdive: Transparently identifying and managing performance interference in virtualized environments. In: Proceedings of USENIX ATC’13: 2013 USENIX Annual Technical Conference, ATC’13. Berkeley, CA, USA: USENIX, pp. 219–230 (2013)

  31. Ferdman, M., Adileh, A., Kocberber, O., Volos, S., Alisafaee, M., Jevdjic, D., Kaynak, C., Popescu, A.D., Ailamaki, A., Falsafi, B.: Clearing the clouds: a study of emerging scale-out workloads on modern hardware. In: ACM SIGARCH Computer Architecture News, vol. 40, no. 1, ACM. NewYork, NY, USA: ACM, pp. 37–48 (2012)

  32. El-Refaey, M.A., Rizkaa, M.A.: Cloudgauge: a dynamic cloud and virtualization benchmarking suite. In: 2010 19th IEEE International Workshop on Enabling Technologies: Infrastructures for Collaborative Enterprises, IEEE. Piscataway, NJ, USA: IEEE, pp. 66–75 (2010)

  33. Silva, M., Hines, M.R., Gallo, D., Liu, Q., Ryu, K.D., da Silva, D.: Cloudbench: Experiment automation for cloud environments. In: Proceedings of the 2013 IEEE International Conference on Cloud Engineering, IC2E’13, IEEE. Piscataway, NJ, USA: IEEE, pp. 302–311 (2013)

  34. Openstack Rally project, https://wiki.openstack.org/wiki/Rally

  35. Fuel: Openstack deployment and management,” Mountain View, CA, USA, http://docs.mirantis.com/openstack/fuel/fuel-5.1/planning-guide.html

Download references

Acknowledgments

This work was supported in part by National Nature Science Foundation of China (61073151, 61572195), Shanghai Agriculture Science Program (2015.3-2), and Shanghai Key Laboratory of Trustworthy Computing Open Project Fund (07dz22304201608).

Author information

Authors and Affiliations

  1. School of Computer Science and Software Engineering, Shanghai Key Laboratory of Trustworthy Computing, East China Normal University, Shanghai, China

    Hongyan Mao

  2. School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China

    Zhengwei Qi & Xinni Ge

  3. Intel APAC R& D Center, Shanghai, China

    Jiangang Duan

Authors
  1. Hongyan Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhengwei Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiangang Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinni Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongyan Mao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mao, H., Qi, Z., Duan, J. et al. Cost-Performance Modeling with Automated Benchmarking on Elastic Computing Clouds. J Grid Computing 15, 557–572 (2017). https://doi.org/10.1007/s10723-017-9412-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-017-9412-4

Keywords

Search

Navigation