Abstract
As one of data mining techniques, outlier detection aims to discover outlying observations that deviate substantially from the reminder of the data. Recently, the Local Outlier Factor (LOF) algorithm has been successfully applied to outlier detection. However, due to the computational complexity of the LOF algorithm, its application to large data with high dimension has been limited. The aim of this paper is to propose grid-based algorithm that reduces the computation time required by the LOF algorithm to determine the k-nearest neighbors. The algorithm divides the data spaces in to a smaller number of regions, called as a “grid”, and calculates the LOF value of each grid. To examine the effectiveness of the proposed method, several experiments incorporating different parameters were conducted. The proposed method demonstrated a significant computation time reduction with predictable and acceptable trade-off errors. Then, the proposed methodology was successfully applied to real database transaction logs of Korea Atomic Energy Research Institute. As a result, we show that for a very large dataset, the grid-LOF can be considered as an acceptable approximation for the original LOF. Moreover, it can also be effectively used for real-time outlier detection.
Highlights
As one of data mining techniques, outlier detection aims to discover outlying observations that deviate substantially from the reminder of the data
The Local Outlier Factor (LOF) algorithm has been successfully applied to outlier detection [3]
The LOF algorithm is a density-based algorithm that detects the local outliers of a dataset by assigning a degree of outlierness, called the local outlier factor (LOF), to each object [4,5]
Summary
As one of data mining techniques, outlier detection aims to discover outlying observations that deviate substantially from the reminder of the data. Identifying an outlier is an important task in many applications because an outlier frequently contains useful information on abnormal behavior in a system, possibly generated by a different mechanism [1,2]. The Local Outlier Factor (LOF) algorithm has been successfully applied to outlier detection [3]. In the LOF algorithm, data points with a lower density than their surrounding points are identified as outliers [3]. As the LOF algorithm can detect “local” outliers regardless of the data distribution of normal behavior [3], it has been applied to various applications including network intrusion detection and process monitoring [6,7].
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have