Performance Assurance Model for HiveQL on Large Data Volume

Abstract

Fast growth of application data has led the migration of existing reporting applications to Big data open source technologies such as Hive and Hadoop. Their wide acceptance also considers their use for servicing on-line analytic queries. Ensuring performance assurance of Hive queries will be required to maintain desired level of application performance. Hive query execution time may increase with increase in data size and change in the cluster size. In this paper, we propose a regression based analytical model to predict execution time of Hive query with growth in data volume. A Hive query is executed as DAG of MapReduce (MR) jobs on Hadoop system, this requires predictive model for MR job execution time. We propose multiple linear regression to compute models for various sub phases of MR job execution and build a consolidated model for predicting the execution time of a MR job on large data volume. We introduce ratio of a phase output record size to its input record size and number of map waves as additional sensitive parameters for predicting MR job execution time. The model is validated with MapReduce benchmark and real world financial application for prediction error within 10 %.