Impact of Node Level Caching in MPI Job Launch Mechanisms

Abstract

The quest for petascale computing systems has seen cluster sizes expressed in terms of number of processor cores increase rapidly. The Message Passing Interface (MPI) has emerged as the defacto standard on these modern, large scale clusters. This has resulted in an increased focus on research into the scalability of MPI libraries. However, as clusters grow in size, the scalability and performance of job launch mechanisms need to be re-visited.

In this work, we study the information exchange involved in the job launch phase of MPI applications. With the emergence of multi-core processing nodes, we examine the benefits of caching information at the node level during the job launch phase. We propose four design alternatives for such node level caches and evaluate their performance benefits. We propose enhancements to make these caches memory efficient while retaining the performance benefits by taking advantage of communication patterns during the job startup phase. One of our cache design – Hierarchical Cache with Message Aggregation, Broadcast and LRU (HCMAB-LRU) reduces the time involved in typical communication stages to one tenth while capping the memory used to a fixed upper bound based on the number of processes. This enables scalable MPI job launching for next generation clusters with hundreds of thousands of processor cores.

This research is supported in part by U.S. Department of Energy grants #DE-FC02-06ER25749 and #DE-FC02-06ER25755; National Science Foundation grants #CNS-0403342, #CCF-0702675 and #CCF-0833169; grant from Wright Center for Innovation #WCI04-010-OSU-0; grants from Intel, Mellanox, Cisco, and Sun Microsystems; Equipment donations from Intel, Mellanox, AMD, Advanced Clustering, Appro, QLogic, and Sun Microsystems.