Workload Shifting: Contention-Insular Disk Arrays for Big Data Systems

Abstract

It is well known that in-place update index, unordered log structured index and ordered log structured index are three typical data organizations which are designed to meet different workload requirements respectively and wildly used in big data storage systems. Differentiated workload requirements in different phase of the data lifecycle, e.g. various types of data are injected into the big data storage systems in the write optimized manner, then they are needed to be read in the read optimized manner for analysis, lead to data organization transformation(data transformation for short). However, the simple mixture of foreground data injection and background data transformation causes serious disk contention. Frequent disk head seeks result in low disk throughput, and not only prolong the data transformation process, but also increase foreground data injection latency. In this paper, we propose \emph{Workload Shifting}, a novel log- structured design that shifts background data transformation away from the foreground data injection. Compared with conventional RAID0 disk array, \emph{Workload Shifting} effectively isolates background data transformation and foreground data injections, avoids the disk contention between them to boost their performance. We have implemented \emph{Workload Shifting} prototype on one multiple disks based disk array. Extensive experimental evaluation results show that compared with conventional RAID0 disk arrays, \emph{Workload Shifting} can avoid disk contention and speed up both data injection and data transformation significantly.