A Performance-Improved and Storage-Efficient Secondary Index for Big Data Processing

Abstract

There are billions of devices in smart grid nowadays. For every data source, it produces thousands of data records every day. Classical relational databases are malfunctioned when dealing with these large-scale data sets. Powerful big data platform is needed to process the information in private clouds of smart grid. HBase is a promising platform to solve these problems. However, finding an effective indexing scheme is still hard because most existing schemes which retrieve data by columns are time-consuming. In this paper, we present a refined secondary index scheme on HBase. It can not only accelerate query process but also save storage space. Experimental results show that when referring to join operation, our proposed indexing scheme provides a minimum 5.584x speedup to a maximum 571.360x speedup compared with a query scheme without any index and it provides a minimum 1.026x speedup to a maximum 4.761x speedup compared with a classical secondary index. Our proposed secondary index scheme is feasible and effective on both query performance and storage efficiency.