Abstract
Nowadays cloud computing has become a popular platform for scientific applications. Cloud computing intends to share a large scale resources and equipments of computation, storage, information and knowledge for scientific researches. Job Scheduling problem is a core and challenging research issue in the current cloud computation area, and the aim is to the reasonable control of the job execution sequence as well as the allocation of computing resources, making the job total completion time of the shortest and resources are fully utilized. Data locality is one of the main factors to influence scheduling algorithm. The paper proposed an improved scheduling algorithm based on priority, after taking full account of data locality (IPDSA), which can distinguish the user’s job levels, so as to reduce the job execution time and avoid losing into locally optimal solution. The experimental results on the Hadoop platform show that the new scheduling algorithm can reduce the job average execution time, and raises the rate availability of resources.
This work was supported by the National Natural Science Foundation of China (61103047), SKLSE Open Foundation (The Open Foundation of State Key Laboratory of Software Engineering, China, and SKLSE 2012-09-18).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Lin, Q.: The cloud computing model based on Hadoop. Modern Computer, 114–116 (2010)
Chemawat, S., Gobioff, H., Leung, S.T.: The Google file system, http://labs.google.com/papers.gfs.html
Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: Proceeding of the 6th Symposium on Operating Systems Design and Implementation (OSDI 2004), pp. 137–150. USENIX Association (2004)
Bansal, S., et al.: Dynamic Task Scheduling in Grid Computing Using Prioritized Round Robin Algorithm. IJCSI International Journal of Computer Science Issues 8(2), 472–477 (2011)
Zaharia, M., Borthakur, D., Sarma, J.S.: Job scheduling for multi-user mapreduce clusters. In: Proceedings of the 5th European Conference IEEE, pp. 145–161 (2009)
Sandholm, T., Lai, K.: Dynamic proportional share scheduling in hadoop. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) JSSPP 2010. LNCS, vol. 6253, pp. 110–131. Springer, Heidelberg (2010)
Tatebe, O., et al.: Grid datafarm architecture for petascale data intensive computing. In: CCGRID 2002. IEEE Computer Society, Washington, DC (2002)
Zaharia, M., Borthakur, D., Sarma, J.S., Elmele-egy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: EuroSys 2010: Proceedings of the 5th European Conference on Computer Systems, pp. 265–278. ACM, New York (2010)
Isard, M., et al.: Quincy: fair scheduling for distributed computing clusters. In: SOSP 2009, pp. 261–276. ACM, New York (2009)
Xie, J., Yin, S., Ruan, X.J., Ding, Z.Y., Tian, Y.: Improving MapReduce performance through data placement in heterogeneous hadoop clusters. In: IEEE International Symposium on Parallel & Distributed Processing, Workshops and PhdForum, pp. 1–9 (2010)
Lin, X., Lu, Y., Deogun, J., Goddard, S.: Real-time divisible load scheduling for cluster computing. In: 13th IEEE Real Time and Embedded Technology and Applications Symposium, RTAS 2007, pp. 303–314, 3–6 (2007)
Yu, J., Buyya, R.: A budget constrained scheduling of workflow applications on utility grids using genetic algorithms. In: Workshop on Workflows in Support of Large-Scale Science, Proceedings of the 15th IEEE International Symposium on High Performance Distributed Computing (HPDC). IEEE CS Press (2006)
Marozzo, F., Talia, D., Trunfio, P.: Adapting MapReduce for Dynamic Environments Using a Peer-to-Peer Model, http://grid.deis.unical.it/papers/pdf/CCA08.pdf
Yang, L., et al.: A new Class of Priority based Weighted Fair Scheduling Algorithm. Physics Procedia 33, 942–948 (2012)
Kyriaki, Z.: Multi-Criteria Job Scheduling in Grid Using an Accelerated Genetic Algorithm. J. Grid Computing 10, 311–323 (2012)
Torabzadeh, E.: Cloud Theory-based Simulated Annealing Approach for Scheduling in the Two-stage Assembly Flowshop. Advances in Engineering Software 41(10), 1243–1258 (2010)
Polo, J., De Nadal, D., Carrera, D., Becerra, Y., Beltran, V., Torres, J., Ayguad´e, E.: Adaptive task scheduling for multi-job mapreduce environments. Technical report UPC-DAC-RR-CAP-2009-28, Departament d’Arquitectura de Com-putadors, Universitat Polit‘ecnica de Catalunya (2009)
Phan, L.T., Zhang, Z., Lo, B.T., Lee, I.: Real-time mapreduce scheduling. Technical Report MS-CIS-10-32, Department of Computer and Information Science, University of Pennsylvania (2010)
Sandholm, T., Lai, K.: Dynamic proportional share scheduling in hadoop. In: Proc. IPDPS Workshops, Atlanta, GA (2010)
Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: 8th USENIX symposium on operating systems design and implementation, pp. 29–42. ACM, New York (2008)
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating mapreduce for multi-core and multiprocessor systems. In: HPCA 2007: Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24. IEEE Computer Society, Washington (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gu, L., Tang, Z., Xie, G. (2014). The Implementation of MapReduce Scheduling Algorithm Based on Priority . In: Li, K., Xiao, Z., Wang, Y., Du, J., Li, K. (eds) Parallel Computational Fluid Dynamics. ParCFD 2013. Communications in Computer and Information Science, vol 405. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-53962-6_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-53962-6_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-53961-9
Online ISBN: 978-3-642-53962-6
eBook Packages: Computer ScienceComputer Science (R0)