Decoupled MapReduce for Shared-Memory Multi-Core Architectures

Abstract

Modern multi-core processors exhibit high integration densities, e.g., up to several tens of cores. Multiple programming frameworks have emerged to facilitate the development of highly parallel applications. The MapReduce programming model, after having demonstrated its usability in the area of distributed computing systems, has been adapted to the needs of shared-memory multi-processors showing promising results in comparison with conventional multi-threaded libraries, e.g., pthreads. In this paper we enhance the traditional MapReduce architecture by decoupling the map and combine phases in order to boost parallel execution. We show that combiners’ memory intensive features limit the system’s degree of parallelism, thus resulting in sub-optimal hardware utilization, leaving space for further performance improvements. The proposed decoupled MapReduce architecture is evaluated into a NUMA server platform, showing that the adoption of the De-MapR runtime enables more efficient hardware utilization and competent run-time improvements. We demonstrate that the proposed solution achieves execution speedups of up to 2.46x compared to a state-of-the-art, shared-memory MapReduce library.