Optimal Design Method of Merging Disk Files Based on Hot Data

Abstract

In recent years, with the rapid development of big data technology, users are more and more inclined to solve the problems of large amount of data and complex business scenarios with big data platform. When the system is faced with the scene of data hotspot and frequent data modification, it will produce a lot of useless data and cause data hotspot problems, which make a lot of access to the level of disk storage. Unreasonable data allocation of HFile will affect I/O performance and reduce system availability. This thesis proposes a comparison strategy based hot data. According to the frequency of data access, this stategy change the selection method of merging subsequence of Exploring Compaction Policy, so as to achieve a more suitable effect for hot data query business.