An effective information detection method for social big data

Abstract

In data mining and knowledge discovery applications, outlier detection is a fundamental problem for robust machine learning and anomaly discovery. There are many successful outlier detection methods, including Local Outlier Factor (LOF), Angle-Based Outlier Factor (ABOF), Local Projection Score (LPS), etc. In this paper, we assume that outliers lie in lower density region and they are at relatively larger distance from any points with a higher local density. In order to identify such outliers quantitatively, the paper proposed a decision graph based outlier detection (DGOD) method. The DGOD method works by firstly calculating the decision graph score (DGS) for each sample, where the DGS is defined as ratio between discriminant distance and local density, next ranking samples according to their DGS values, and finally, returning samples with top-r largest DGS values as outliers. Experimental results on synthetic and real-world datasets have confirmed its effectiveness on outlier detection problems, and it is a general and effective information detection method, which is robust to data shape and dimensionality.