Transparent many‐core partitioning for high‐performance big data I/O

Abstract

SummaryAs the number of cores equipped in a single computing node is rapidly increasing, utilizing many cores for contemporary applications in an efficient manner is a challenging issue. We need to consider both parallelization and locality to fully exploit many cores for multifarious operations of emerging applications. In particular, big data applications perform computation and I/O intensive operations alternately. For instance, Apache Hadoop MapReduce assumes local persistent storage for each computing node. Thus, unlike traditional parallel programming models, the MapReduce framework performs not only networking but also storage I/O. In this study, we aim to improve the locality of network and storage I/O operations on many‐core systems by partitioning cores for I/O system calls and event handlers. In order to implement fine‐grained many‐core partitioning, we decouple the system call context from the user‐level process by suggesting message‐based system calls. The suggested design provides user‐level transparency and does not require any kernel‐level modifications. In addition, we propose a scheme that dynamically decides the core affinity of system calls and event handlers by considering locality, run‐time loads, and hardware architectures. The experimental results show that the proposed many‐core partitioning can improve the locality of network and storage I/O operations in an integrated manner for MapReduce applications.