Not All Joules are Equal: Towards Energy-Efficient and Green-Aware Data Processing Frameworks

Abstract

Interests have been growing in integrating renewableenergy into data centers, which attracts many researchefforts in developing green-aware algorithms and systems. However, little attention was paid to the efficiency of each jouleconsumed by data center workloads. In fact, not all joulesare equal in the sense that the amount of work that can bedone by a joule can vary significantly in data centers. Ignoringthis fact leads to significant energy waste (by 25% of the totalenergy consumption in Hadoop YARN on a Facebook productiontrace according to our study). In this paper, we investigatehow to exploit such joule efficiency to maximize the benefitsof renewable energy for MapReduce framework. We developjob/task scheduling algorithms with a particular focus on thefactors on joule efficiency in the data center, including theenergy efficiency of MapReduce workloads, renewable energysupply and the battery usage. We further develop a simpleyet effective performance-energy consumption model to guideour scheduling decisions. We have implemented GreenMR, anenergy-efficient and green-aware MapReduce framework, on topof Hadoop YARN. The experiments demonstrate the accuracyof our models, and the effectiveness of our energy-efficient andgreen-aware optimizations over Hadoop YARN and a state-of-the-art green-aware Hadoop YARN implementation.