A Grid Partition-Based Local Outlier Factor by Reachability Distance for Data Stream Processing

Abstract

In the field of big data, outlier detection is an important issue in many applications, such as communications, health, and network intrusion detection. Data streams present a challenge to the traditional outlier detection methods due to their unique characteristics, i.e., large volume and sequential structure. The Local Outlier Factor (LOF) is a popular algorithm used in anomaly detection to find outliers in a data stream. However, the LOF algorithm has a drawback. When a new point is added to the data stream, the algorithm needs to reprocess the measurement from the beginning of any change in the dataset. Additionally, LOF requires the whole dataset to be stored in memory. These issues were solved in the Grid Partition-based Local Outlier Factor (GP-LOF) algorithm. A new algorithm is introduced in this paper to improve the GP-LOF algorithm. It is based on the reachability distance measurement in LOF, and we name it Grid Partition-based Local Outlier Factor by Reachability distance (GP-LOFR) algorithm. GP-LOFR has slightly improved the accuracy rate of local outlier detection in our experiments with several real-world datasets.