Distributed Dynamic Cuckoo Filter System Based on Redis Cluster

Abstract

With the exponential growth of network data storage scale, the issue of uniform distribution and efficient retrieval of data in the distributed storage systems such as the Redis cluster has received increasing attention in recent years. In view of the existing problems in scalability, usability and other aspects of the solution in the current research, we propose the Distributed Dynamic Cuckoo Filter system based on Redis cluster. On the one hand, we introduce an efficient hash index structure—Dynamic Cuckoo Filter, which only stores the fingerprint information of data, and automatically scalable capacity to meet the demand of data storage on a dynamic scale. On the other hand, we use the consistent hashing algorithm to construct Redis cluster, and use the thorough communication mechanism of Redis cluster to achieve the data sharing and efficient utilization of multi-machine filters. The scheme proposed in this paper can take the time and space efficiency into account, greatly improve the retrieval performance of massive data, and improve the reliability and availability of Redis cluster.