Existing Outlier Detection Methods Research Articles

Robust support vector data description for outlier detection with noise or uncertain data

As an example of one-class classification methods, support vector data description (SVDD) offers an opportunity to improve the performance of outlier detection and reduce the loss caused by outlier occurrence in many real-world applications. However, due to limited outliers, the SVDD model is built only by using the normal data. In this situation, SVDD may easily lead to over fitting when the normal data contain noise or uncertainty. This paper presents two types of new SVDD methods, named R-SVDD and εNR-SVDD, which are constructed by introducing cutoff distance-based local density of each data sample and the ε-insensitive loss function with negative samples. We have demonstrated that the proposed methods can improve the robustness of SVDD for data with noise or uncertainty by extensive experiments on ten UCI datasets. The experimental results have shown that the proposed εNR-SVDD is superior to other existing outlier detection methods in terms of the detection rate and the false alarm rate. Meanwhile, the proposed R-SVDD can also achieve a better outlier detection performance with only normal data. Finally, the proposed methods are successfully used to detect the image-based conveyor belt fault.

Entropy-based outlier detection using semi-supervised approach with few positive examples

Outlier detection is an important problem in data mining that aims to discover useful exceptional and unusual patterns hidden in large data sets. Fraud detection, time series monitoring, intrusion detection and medical condition monitoring are some of the most common applications of outlier detection. Most existing outlier detection methods are based on supervised or unsupervised learning while some others use semi-supervised approaches. However, in many real world applications, there are not enough labeled data for training and only a few positive labeled samples are available. This paper presents an entropy-based solution. The proposed method consists of two phases. First, reliable negative examples are extracted from positive and unlabeled data and then, as the second phase, the entropy-based outlier detection algorithm is employed to detect top N outliers. Many experiments on real and synthetic data sets are performed. The experimental results on synthetic and real data demonstrated superiority of the proposed outlier detection method in comparison of unsupervised state-of-the-art outlier detection strategies.

MAXIMUM VOLUME OUTLIER DETECTION AND ITS APPLICATIONS IN CREDIT RISK ANALYSIS

Because of the scarcity and diversity of outliers, it is very difficult to design a robust outlier detector. In this paper, we first propose to use the maximum margin criterion to sift unknown outliers, which demonstrates superior performance. However, the resultant learning task is formulated as a Mixed Integer Programming (MIP) problem, which is computationally hard. Therefore, we alter the recently developed label generating technique, which efficiently solves a convex relaxation of the MIP problem of outlier detection. Specifically, we propose an effective procedure to find a largely violated labeling vector for identifying rare outliers from abundant normal patterns, and its convergence is also presented. Then, a set of largely violated labeling vectors are combined by multiple kernel learning methods to robustly detect outliers. Besides these, in order to further enhance the efficacy of our outlier detector, we also explore the use of maximum volume criterion to measure the quality of separation between outliers and normal patterns. This criterion can be easily incorporated into our proposed framework by introducing an additional regularization. Comprehensive experiments on toy and real-world data sets verify that the outlier detectors using the two proposed criteria outperform existing outlier detection methods. Furthermore, our models are employed to detect corporate credit risk and demonstrate excellent performance.

Identification of Outliers in Oxazolines and Oxazoles High Dimension Molecular Descriptor Dataset Using Principal Component Outlier Detection Algorithm and Comparative Numerical Study of Other Robust Estimators

From the past decade outlier detection has been in use. Detection of outliers is an emerging topic and is having robust applications in medical sciences and pharmaceutical sciences. Outlier detection is used to detect anomalous behaviour of data. Typical problems in Bioinformatics can be addressed by outlier detection. A computationally fast method for detecting outliers is shown, that is particularly effective in high dimensions. PrCmpOut algorithm make use of simple properties of principal components to detect outliers in the transformed space, leading to significant computational advantages for high dimensional data. This procedure requires considerably less computational time than existing methods for outlier detection. The properties of this estimator (Outlier error rate (FN), Non-Outlier error rate(FP) and computational costs) are analyzed and compared with those of other robust estimators described in the literature through simulation studies. Numerical evidence based Oxazolines and Oxazoles molecular descriptor dataset shows that the proposed method performs well in a variety of situations of practical interest. It is thus a valuable companion to the existing outlier detection methods.

Local Distance-based Outlier Removal for Scattered data through Minimum Spanning Tree

Minimum spanning tree-based algorithm is capable of detecting outlier from data set having irregular shape, size and boundaries. Detecting outliers using Minimum Spanning Tree-based algorithm is a big desire. Outlier detection is an extremely important task in a wide variety of application. Outliers have little or no influence, and may be isolated as noise in the data set. Existing outlier detection methods are ineffective on scattered data set due to implicit data patterns and parameter setting issues. In this paper we define a novel Local Distance-based Outlier Factor (LDOF) to measure the outlier-ness of objects in the scattered data sets. Based on the LDOF computed, we propose a new method to detect outliers present in the scattered data set.

Pricing fraud detection in online shopping malls using a finite mixture model

Although pricing fraud is an important issue for improving service quality of online shopping malls, research on automatic fraud detection has been limited. In this paper, we propose an unsupervised learning method based on a finite mixture model to identify pricing frauds. We consider two states, normal and fraud, for each item according to whether an item description is relevant to its price by utilizing the known number of item clusters. Two states of an observed item are modeled as hidden variables, and the proposed models estimate the state by using an expectation maximization (EM) algorithm. Subsequently, we suggest a special case of the proposed model, which is applicable when the number of item clusters is unknown. The experiment results show that the proposed models are more effective in identifying pricing frauds than the existing outlier detection methods. Furthermore, it is presented that utilizing the number of clusters is helpful in facilitating the improvement of pricing fraud detection performances.

Detecting outlying subspaces for high-dimensional data: the new task, algorithms, and performance

In this paper, we identify a new task for studying the outlying degree (OD) of high-dimensional data, i.e. finding the subspaces (subsets of features) in which the given points are outliers, which are called their outlying subspaces. Since the state-of-the-art outlier detection techniques fail to handle this new problem, we propose a novel detection algorithm, called High-Dimension Outlying subspace Detection (HighDOD), to detect the outlying subspaces of high-dimensional data efficiently. The intuitive idea of HighDOD is that we measure the OD of the point using the sum of distances between this point and itsknearest neighbors. Two heuristic pruning strategies are proposed to realize fast pruning in the subspace search and an efficient dynamic subspace search method with a sample-based learning process has been implemented. Experimental results show that HighDOD is efficient and outperforms other searching alternatives such as the naive top–down, bottom–up and random search methods, and the existing outlier detection methods cannot fulfill this new task effectively.