Double locality sensitive hashing Bloom filter for high-dimensional streaming anomaly detection

Abstract

Most of the existing large-scale high-dimensional streaming anomaly detection methods suffer from extremely high time and space complexity. Moreover, these models are very sensitive to parameters,make their generalization ability very low, can also be merely applied to very few specific application scenarios. This paper proposes a three-layer structure high-dimensional streaming anomaly detection model, which is called the double locality sensitive hashing Bloom filter, namely dLSHBF. We first build the former two layers that is double locality sensitive hashing (dLSH), proving that the dLSH method reduces the hash coding length of the data, and it ensures that the projected data still has a favorable mapping distance-preserving property after projection. Second, we use a Bloom filter to build the third layer of dLSHBF model, which used to improve the efficiency of anomaly detection. Six large-scale high-dimensional data stream datasets in different IIoT anomaly detection domains were selected for comparison experiments. First, extensive experiments show that the distance-preserving performance of the former dLSH algorithm proposed in this paper is significantly better than the existing LSH algorithms. Second, we verify the dLSHBF model more efficient than the other existing advanced Bloom filter model (for example Robust Bloom Filter, Fly Bloom Filter, Sandwich Learned Bloom Filter, Adaptive Learned Bloom Filters). Compared with the state of the art, dLSHBF can perform with the detection rate (DR) and false alarm rate (FAR) of anomaly detection more than 97%, and less than 2.2% respectively. Its effectiveness and generalization ability outperform other existing streaming anomaly detection methods.