Scheduling Distributed Clusters of Parallel Machines : Primal-Dual and LP-based Approximation Algorithms

Abstract

The Map-Reduce computing framework rose to prominence with datasets of such size that dozens of machines on a single cluster were needed for individual jobs. As datasets approach the exabyte scale, a single job may need distributed processing not only on multiple machines, but on multiple clusters. We consider a scheduling problem to minimize weighted average completion time of n jobs on m distributed clusters of parallel machines. In keeping with the scale of the problems motivating this work, we assume that (1) each job is divided into m “subjobs” and (2) distinct subjobs of a given job may be processed concurrently. When each cluster is a single machine, this is the NP-Hard concurrent open shop problem. A clear limitation of such a model is that a serial processing assumption sidesteps the issue of how different tasks of a given subjob might be processed in parallel. Our algorithms explicitly model clusters as pools of resources and effectively overcome this issue. Under a variety of parameter settings, we develop two constant factor approximation algorithms for this problem. The first algorithm uses an LP relaxation tailored to this problem from prior work. This LP-based algorithm provides strong performance guarantees. Our second algorithm exploits a surprisingly simple mapping to the special case of one machine per cluster. This mapping-based algorithm is combinatorial and extremely fast. These are the first constant factor approximations for this problem.

Appendix: A Reduction for Minimizing Total Weighted Lateness on Identical Parallel Machines

The problem of minimizing total weighted lateness on a bank of identical parallel machines is typically denoted $P || \sum w_jL_j$, where the lateness of a job with deadline $d_j$ is $L_j \doteq \max {\{C_j - d_j, 0\}}$. The reduction we offer below shows that $P || \sum w_j L_j$ can be stated in terms of $CC || \sum w_jC_j$ at optimality. Thus while a $\Delta$ approximation to $CC || \sum w_jC_j$ does not imply a $\Delta$ approximation to $P || \sum w_j L_j$, the reduction below nevertheless provides new insights on the structure of $P || \sum w_j L_j$.

Definition 17

(Total Weighted Lateness Reduction) Let $I = (p, d, w, m)$ denote an instance of $P || \sum w_j L_j$. p is the set of processing times, d is the set of deadlines, w is the set of weights, and m is the number of identical parallel machines. Given these inputs, we transform $I \in \Omega _{P || \sum w_j L_j}$ to $I' \in \Omega _{CC}$ in the following way.

Create a total of $n + 1$ clusters. Cluster 0 has m machines. Job j has processing time $p_j$ on this cluster, and $|T_{j0}| = 1$. Clusters 1 through n each consist of a single machine. Job j has processing time $d_j$ on cluster j, and zero on all clusters other than cluster 0 and cluster j. Denote this problem $I'$.

We refer the reader to Fig. 2 for an example output of this reduction.

Theorem 18

Let I be an instance of $P || \textstyle \sum w_j L_j$. Let $I'$ be an instance of $CC|| \sum w_j C_j$ resulting from the transformation described above. Any list schedule $\sigma$ that is optimal for $I'$ is also optimal for I.

Proof

If we restrict the solution space of $I'$ to single permutations (which we may do without loss of generality), then any schedule $\sigma$ for I or $I'$ produces the same value of $\sum _{j \in N} w_j(C_j - d_j)^+$ for I and $I'$. The additional clusters we added for $I'$ ensure that $C_j \ge d_j$. Given this, the objective for I can be written as $\sum _{j \in N} w_j d_j + w_j(C_j - d_j)^+$. Because $w_j d_j$ is a constant, any permutation to solve $I'$ optimally also solves $\sum _{j \in N} w_j (C_j - d_j)^+$ optimally. Since $\sum _{j \in N} w_j (C_j - d_j)^+ = \sum _{j \in N} w_j L_j$, we have the desired result. $\square$