A scalable Map Reduce tasks scheduling: a threading-based approach

Abstract

The Map Reduce paradigm is now considered a standard platform that is used for large-scale data processing and management. A major operation that the Map Reduce platform relies on greatly is tasks scheduling. Although many schedulers have been presented, task scheduling is still one of the major problems that face Map Reduce frameworks. Schedulers need to maintain data locality to achieve an acceptable performance by avoiding several data transmissions. Hence, in this paper, we propose a new scheduling algorithm named 'MTL' that utilises multi-threading principles. The MTL scheduler assigns a dedicated thread for each data block. Indeed, the multi-threading approach shows great results that make our MTL scheduler a scalable one that performs well. At the same time, it maintains the locality property. During the evaluation of the MTL scheduler performance, two main factors were taken into consideration; the simulation time and the energy consumption. The MTL scheduler is then compared with other existing schedulers such as FIFO, matchmaking, and delay schedulers. The MTL scheduler showed favourable results and proved its advantages over other existing schedulers.