MarVeLScaler: A Multi-View Learning-Based Auto-Scaling System for MapReduce

Abstract

To promote cloud computing from current pay-per-request model to truly pay-per-use, tenants are crying for automatic tools to auto-estimate the amount of resources for MapReduce jobs. Such tools call for accurately quantifying the relationship among workload, resources and completion time. Various prediction models have been proposed. However, none of these models takes virtual machines&#x2019; (VMs) performance variance during a job&#x0027;s execution into consideration, leading to underestimate the required resources and exceed the job&#x0027;s deadline. To address this problem, we propose a multi-view deep learning model to capture real-time performance variance and automatically scale out the cloud cluster whenever necessary. We implement <i>MarVeLScaler</i>, a prototype system including two useful modules, namely, <i>Scale Estimator</i> and <i>Scale Controller</i>. Scale Estimator preliminarily estimates the required cluster size for a MapReduce job with given a concrete workload and deadline. During the runtime, Scale Controller adjusts the scale of the cluster according to its real-time running status to guarantee the job finished on time. We evaluate the performance of MarVeLScaler based on Hadoop in Alibaba Cloud. Experiments show that MarVeLScaler can provide 98.4 percent accuracy of prediction in determining initial cluster size, and save 30.8 percent of expense while still guaranteeing similar performance compared with the state-of-the-art methods.