A Sliding-Window Algorithm Implementation in MapReduce

Abstract

A limited resource processing platform may not be suited to process a large volume of data. The distributed processing platforms can solve this problem by incorporating commodity hardware collaboratively to process a large volume of data. The MapReduce programming framework is one candidate framework for large-scale processing, and Hadoop is its open-source implementation. This framework consists of the Hadoop Distributed File System and the MapReduce for computation capabilities. However, the MapReduce framework does not allow for data sharing for computation among the computing nodes. In this paper, we present an implementation of a sliding-window algorithm for data sharing for computation dependency in MapReduce. The algorithm is designed to facilitate the data processing a sequential order, e.g., moving average. The algorithm utilizes the MapReduce job metadata, e.g., input split size, to prepare the shared data between the computing nodes without violating the MapReduce fault tolerance handling mechanism.

Appendices

Appendix A: Java Class for Record Sharing—Mapper Class

public class RShareMap extends Mapper<LongWritable, Text, LongWritable, Text> { @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { Configuration conf = context.getConfiguration(); int SplitSize = Integer.parseInt(conf.get("SplitSize")); InputSplit split = context.getInputSplit(); long SplitLength = ((FileSplit) split).getLength(); long SplitOffset = ((FileSplit) split).getStart(); // Split the lines so you can select the element you want to copy forward String line = value.toString(); System.out.println(line.length()); // Split up each element of the line String[] elements = line.split("\n"); int window = 2; // number of records to share with the next map int size = 6;//one character size. Text outputValue = new Text(); LongWritable OutKey = new LongWritable(); long idx1 = SplitLength-(window-1)*size; // used for first split only SplitOffset =0 long idx2 = idx1+(window-1)*size;// used for first split only SplitOffset =0 /// Split values if ((SplitOffset==0)&&(key.get()>=idx1)& &(key.get()<=idx2)){ int temp= (int) Math.log10( ((window-1) +SplitLength)*size ); // calculate the new space for the copied forward element for first split only temp=temp+1; int Space = (int)Math.pow(10, temp); outputValue.set(elements[0]); long nkey= (long) (SplitLength*Space); nkey=nkey+key.get(); OutKey.set(nkey); context.write(OutKey, outputValue); } // End of First split values // Subsequent splits except the last one long nkey=key.get(); int x= (int) Math.log10( ((window-1) +SplitLength)*size ); x=x+1; int NewSpace = (int)Math.pow(10, x); long SplitNew=SplitOffset*NewSpace; if(SplitOffset>0){ long idx3 = SplitOffset+SplitLength-(window-1)*size; // used for any split other than SplitOffset =0 long idx4 = idx3+(window-1)*size; // used for any split other than SplitOffset =0 boolean test = (nkey>idx3)&&(nkey<=idx4)&& (SplitLength==SplitSize); if(test){ outputValue.set(elements[0]); long tempkey = (SplitOffset+SplitLength) *NewSpace; long tKey = tempkey+nkey; OutKey.set(tKey); context.write(OutKey, outputValue); //System.out.println(SplitLength); } } long newKey = SplitNew+key.get(); context.write(new LongWritable(newKey), value); } }

Appendix B : Java Class for Record Sharing—Reducer Class

public class RShareReduce extends Reducer<LongWritable, Text, LongWritable, Text> { public void reduce(LongWritable key, Text values, Context context) throws IOException, InterruptedException { long k1= key.get()+100; context.write(new LongWritable (k1), values); } }

Appendix C: Java Class for Record Sharing—Driver Class

public class RShareDriver { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); // Use programArgs array to retrieve program arguments. String[] programArgs = new GenericOptionsParser (conf, args).getRemainingArgs(); conf.set("SplitSize", "12"); Job job = new Job(conf); job.setJarByClass(RShareDriver.class); job.setMapperClass(RShareMap.class); job.setReducerClass(RShareReduce.class); job.setOutputKeyClass(LongWritable.class); job.setOutputValueClass(Text.class); FileInputFormat.setMaxInputSplitSize(job, 12); // change the max split FileInputFormat.setMinInputSplitSize(job, 12); // change the min split // TODO: Update the input path for the location of the inputs of the map-reduce job. FileInputFormat.addInputPath(job, new Path(programArgs[0])); // TODO: Update the output path for the output directory of the map-reduce job. FileOutputFormat.setOutputPath(job, new Path(programArgs[1])); // Submit the job and wait for it to finish. job.waitForCompletion(true); // Submit and return immediately: // job.submit(); } }