Skip to main content
SpringerLink
Log in

Shapp: Workload Management System for Massive Distributed Calculations

  • Conference paper
  • First Online:
Software Engineering Methods in Intelligent Algorithms (CSOC 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 984))

Included in the following conference series:

Abstract

The paper presents an overview of existing workload management systems and diagnoses difficulties in their use. Such systems require a complex configuration, and the process of ordering tasks to be carried out is burdened with many restrictions. A new solution is presented which supports the use of massive distributed computations. The work presents the process of designing, implementing and testing the workload management Shapp library, based on the HTCondor system. It implements a convenient application interface in the form of a dynamically linked library, which extends the capabilities of existing applications with a convenient mechanism allowing the use of massive distributed processing. Recursive computations in tree-like structure are possible using Shapp library.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Daszczuk, W.B., Mieścicki, J., Grabski, W.: Distributed algorithm for empty vehicles management in personal rapid transit (PRT) network. J. Adv. Transp. 50(4), 608–629 (2016). https://doi.org/10.1002/atr.1365

    Article  Google Scholar 

  2. Daszczuk, W.B.: Discrete event simulation of personal rapid transit (PRT) systems. Autobusy-TEST 17(3), 1302–1310 (2016). arXiv:1705.05237

  3. Sfiligoi, I.: glideinWMS—a generic pilot-based workload management system. J. Phys. Conf. Ser. 119(6), 062044 (2008). https://doi.org/10.1088/1742-6596/119/6/062044

    Article  Google Scholar 

  4. Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: Buyya, R. (ed.) Fifth IEEE/ACM International Workshop on Grid Computing, Pittsburgh, PA, 8 November 2004, pp. 4–10. IEEE (2004). https://doi.org/10.1109/grid.2004.14

  5. Aad, G.: The ATLAS experiment at the CERN large hadron collider. J. Instrum. 3, 407 (2008). https://iopscience.iop.org/article/10.1088/1748-0221/3/08/S08003/meta

  6. Méndez, B.J.H.: SpaceScience@Home: authentic research projects that use citizen scientists. In: Garmany, C., Gibbs, M.G., Moody, J.W. (eds.) EPO and a Changing World: Creating Linkages and Expanding Partnerships, Chicago, IL 5–7 September 2007, pp. 219–226. ASP Press, San Francisco (2008). http://adsabs.harvard.edu/full/2008ASPC..389..219M

  7. Patoli, M.Z., Gkion, M., Al-Barakati, A., Zhang, W., Newbury, P., White, M.: An open source Grid based render farm for Blender 3D. In: 2009 IEEE/PES Power Systems Conference and Exposition, Seattle, WA, 15–18 March 2009, pp. 1–6. IEEE (2009). https://doi.org/10.1109/psce.2009.4839978

  8. Czarnul, P., Kuchta, J., Matuszek, M.: Parallel computations in the volunteer–based comcute system. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds.) International Conference on Parallel Processing and Applied Mathematics, PPAM 2013, Warsaw, Poland, 8–11 September 2013. LNCS, vol. 8384, pp. 261–271. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-55224-3_25

    Chapter  Google Scholar 

  9. HTCondor. https://research.cs.wisc.edu/htcondor/

  10. Pool, M.: distcc, a fast free distributed compiler. In: The Linux Conference, Las Vegas, NV, June 2004, pp. 1879–1885 (2004). https://fossies.org/linux/distcc/doc/web/distcc-lca-2004.pdf

  11. Zhang, W.: Linux virtual server for scalable network services. In: Ottawa Linux Symposium, Ottawa, Canada, 22 July 2000, pp. 1–10 (2000). www.linuxvirtualserver.org/ols/lvs.pdf

  12. Owsiany, M.: High availability in Linux System (in Polish: Wysoka dostępność w systemie Linux) (2003). http://marcin.owsiany.pl/studia/inf-4_rok/swn/referat.pdf

  13. Cassen, A.: Keepalived. http://www.keepalived.org/

  14. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: Cappello, F., Wang, C.-L., Buyya, R. (eds.) 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, Shanghai, China, 18–21 May 2009, pp. 124–131. IEEE (2009). https://doi.org/10.1109/ccgrid.2009.93

  15. Raman, R., Livny, M., Solomon, M.: Matchmaking: distributed resource management for high throughput computing. In: Seventh International Symposium on High Performance Distributed Computing, Chicago, IL, 31 July 1998, pp. 140–146. IEEE (1998). https://doi.org/10.1109/hpdc.1998.709966

  16. Santos, A., Almeida, F., Blanco, V.: Lightweight web services for high performance computing. In: Oquendo, F. (ed.) European Conference on Software Architecture, Aranjuez, Spain, 24–26 September 2007. LNCS, vol. 4758, pp. 225–236. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-75132-8_18

  17. Raicu, I., Foster, I.T., Zhao, Y.: Many-task computing for grids and supercomputers. In: Workshop on Many-Task Computing on Grids and Supercomputers, Austin, TX, 17 November 2008, pp. 1–11. IEEE (2008). https://doi.org/10.1109/mtags.2008.4777912

  18. Satyanarayana, K.C., Gani, R., Abildskov, J.: Polymer property modeling using grid technology for design of structured products. Fluid Phase Equilib. 261(1–2), 58–63 (2007). https://doi.org/10.1016/j.fluid.2007.07.058

    Article  Google Scholar 

  19. Zakrzewska, K., Bouvier, B., Michon, A., Blanchet, C., Lavery, R.: Protein–DNA binding specificity: a grid-enabled computational approach applied to single and multiple protein assemblies. Phys. Chem. Chem. Phys. 11(45), 10712 (2009). https://doi.org/10.1039/b910888m

    Article  Google Scholar 

  20. Bird, I.: Computing for the Large Hadron Collider. Annu. Rev. Nucl. Part. Sci. 61(1), 99–118 (2011). https://doi.org/10.1146/annurev-nucl-102010-130059

    Article  Google Scholar 

  21. Raicu, I., Foster, I., Zhao, Y., Szalay, A., Little, P., Moretti, C.M., Chaudhary, A., Thain, D.: Towards data intensive many-task computing. In: Kosar, T. (ed.) Data Intensive Distributed Computing: Challenges and Solutions for Large-Scale Information Management, pp. 28–73. IGI Global (2012). https://doi.org/10.4018/978-1-61520-971-2.ch002

  22. Nishimura, H., Timossi, C.: Mono for cross-platform control system environment. In: 6th International Workshop on Personal Computers and Particle Accelerator Controls, Newport News, VA, 24–27 September 2006 (2006). https://escholarship.org/uc/item/3hn297s0

  23. Kolici, V., Herrero, A., Xhafa, F.: On the performance of oracle grid engine queuing system for computing intensive applications. J. Inf. Process. Syst. 10(4), 491–502 (2014). https://doi.org/10.3745/JIPS.01.0004

    Article  Google Scholar 

  24. Foster, I., Kesselman, C.: Globus: a metacomputing infrastructure toolkit. Int. J. Supercomput. Appl. High Perform. Comput. 11(2), 115–128 (1997). https://doi.org/10.1177/109434209701100205

    Article  Google Scholar 

  25. Krieger, M.T., Torreno, O., Trelles, O., Kranzlmüller, D.: Building an open source cloud environment with auto-scaling resources for executing bioinformatics and biomedical workflows. Futur. Gener. Comput. Syst. 67, 329–340 (2017). https://doi.org/10.1016/j.future.2016.02.008

    Article  Google Scholar 

  26. Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) Job Scheduling Strategies for Parallel Processing, pp. 44–60. Springer, Heidelberg (2003). https://doi.org/10.1007/10968987_3

    Chapter  Google Scholar 

  27. Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurr. Comput. Pract. Exp. 17(2–4), 323–356 (2005). https://doi.org/10.1002/cpe.938

    Article  Google Scholar 

  28. Asagba, P., Ogheneovo, E.: Qualities of grid computing that can last for ages. J. Appl. Sci. Environ. Manag. 12(4) (2010). https://doi.org/10.4314/jasem.v12i4.55218

  29. Georgatos, F., Gkamas, V., Ilias, A., Kouretis, G., Varvarigos, E.: A grid-enabled CPU scavenging architecture and a case study of its use in the greek school network. J. Grid Comput. 8(1), 61–75 (2010). https://doi.org/10.1007/s10723-009-9143-2

    Article  Google Scholar 

  30. Galecki, T.: The environment of support of a massive distributed computing (in Polish: Srodowisko wsparcia masowego przetwarzania rozproszonego), BSc thesis, Warsaw University of Technology, Institute of Computer Science, 50p. (2019). http://repo.bg.pw.edu.pl/index.php/pl/r#/info/bachelor/WUTcac04f4e732f434590a18a4b4d6fcf68/?r=diploma&tab=&lang=pl

  31. Ossher, H., Kaplan, M., Harrison, W., Katz, A., Kruskal, V.: Subject-oriented composition rules. ACM SIGPLAN Not. 30(10), 235–250 (1995). https://doi.org/10.1145/217839.217864

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computer Science, Warsaw University of Technology, Nowowiejska Street 15/19, 00-665, Warsaw, Poland

    Tomasz Gałecki & Wiktor B. Daszczuk

Authors
  1. Tomasz Gałecki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wiktor B. Daszczuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wiktor B. Daszczuk .

Editor information

Editors and Affiliations

  1. Faculty of Applied Informatics, Tomas Bata University in Zlín, Zlín, Czech Republic

    Radek Silhavy

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gałecki, T., Daszczuk, W.B. (2019). Shapp: Workload Management System for Massive Distributed Calculations. In: Silhavy, R. (eds) Software Engineering Methods in Intelligent Algorithms. CSOC 2019. Advances in Intelligent Systems and Computing, vol 984. Springer, Cham. https://doi.org/10.1007/978-3-030-19807-7_31

Download citation

Publish with us

Policies and ethics

Search

Navigation