Financial Big Data Hot and Cold Separation Scheme Based on HBase and Redis

Abstract

In the era of big data, HBase has been widely used in scenarios of massive unstructured data. For the financial big data, due to the integrity and timing of it, unreasonable data storage and management usually lead to hot spots that decreases the query performance. In practice, the separation of hot and cold financial data will improve data query performance and utilization rate of cluster resources. In this paper, a hot and cold data separation scheme is designed, to store infrequently queried financial data to HBase, and frequently queried one to Redis. The cold data is reasonably planned and managed through pre-partitioning and row key design for HBase. A hot data cache based on Redis is realized to improve the query speed and reduces the pressure of HBase. In addition, due to the lack of Redis's inherent cache elimination strategy, we propose a caching strategy based on the frequencies of updating and querying operations. The experimental results show that the scheme can effectively avoid the hot storage problem, and improve the query performance, and improve the cache hit ratio of Redis. Therefore, the number of cold data access requests can be effectively reduced.