A reinforcement learning approach to map reduce auto-configuration under networked environment

Abstract

Listen

Translate article

Hadoop-an open-source implementation of MapReduce is widely used for distributed processing model of large-scale data-intensive applications, configuration is crucial to the performance of MapReduce. As they expect that end users determine appropriate MapReduce parameters for running a job, which require in-depth knowledge of system and may lead to performance degradation. We propose a reinforcement learning approach to enable automated tuning configuration of MapReduce parameters, the RL approach has an initialisation policy with offline learning to reduce online learn time in different circumstance. Experimental results demonstrate that the approach can auto-configure the system, have better computers performance and shorter running time.