KVBTree: A Key/Value Based Storage Structure for Large-Scale Electric Power Data

Abstract

There are lots of data communications among subtasks in big data processing, especially those key pair intermediate data. Its a key point concerning big data analytic area to process those intermediate data efficiently. However state of the art big data systems rare put an eye on the optimization and effeicient management of those intermediate data. On one hand, intermediate data in MapReduce and other batch systems are stored sequentially in files. Those intermediate date cannot be merged automatically, extra shuffle phase is needed in MapReduce to aggregate values with the same key, which surely costs extra time. On the other hand, some distributed message management system such as stream data processing system can aggregate the intermediate messages effectively, but they are at a much higher-level abstraction. Those systems usually suffer from complicated structure, which may incur the performance degradation. In this paper, we propose a B+ tree based data structure, KVBTree, to manage the intermediate data effectively. An efficient cache strategy is then proposed to increase the storage capability of the KVBTree. We also present the strategies for the querying, inserting and traversing of the data elements in KVBTree. We evaluate the time complexity and the space complexity of KVBTree in the final part of this paper. The experimental results show that KVBTree is able to play an effective role concerning the electric power data.