Existing Outlier Detection Methods Research Articles

Boundary-aware local Density-based outlier detection

Outlier detection is crucial for improving the performance of machine learning algorithms and is particularly vital in data sets possessing a small number of points. While the existing outlier detection methods deliver good results on a certain data set, the results are rather down on some data sets. Besides all these aspects, there is also a need for an algorithm that quickly processes high-dimensional data sets. To satisfy these requirements, we propose an unsupervised local outlier detection method that can draw the neighborhood boundaries of the data points via Chebyshev inequality. The proposed method sets the boundaries of the points through the so-called deviation parameter that correlates to the standard deviation of the data distribution and then detects outliers by quantifying their neighborhood densities. The experimental results on real-world and synthetic data sets show the efficacy of the proposed method in comparison to the state-of-the-art methods. The source code of the proposed algorithm and the data sets are at https://github.com/fatihaydin1/BLDOD.

Benchmarking Outlier Detection Methods for Detecting IEM Patients in Untargeted Metabolomics Data.

Untargeted metabolomics (UM) is increasingly being deployed as a strategy for screening patients that are suspected of having an inborn error of metabolism (IEM). In this study, we examined the potential of existing outlier detection methods to detect IEM patient profiles. We benchmarked 30 different outlier detection methods when applied to three untargeted metabolomics datasets. Our results show great differences in IEM detection performances across the various methods. The methods DeepSVDD and R-graph performed most consistently across the three metabolomics datasets. For datasets with a more balanced number of samples-to-features ratio, we found that AE reconstruction error, Mahalanobis and PCA reconstruction error also performed well. Furthermore, we demonstrated the importance of a PCA transform prior to applying an outlier detection method since we observed that this increases the performance of several outlier detection methods. For only one of the three metabolomics datasets, we observed clinically satisfying performances for some outlier detection methods, where we were able to detect 90% of the IEM patient samples while detecting no false positives. These results suggest that outlier detection methods have the potential to aid the clinical investigator in routine screening for IEM using untargeted metabolomics data, but also show that further improvements are needed to ensure clinically satisfying performances.

Outlier detection algorithm based on k-nearest neighbors-local outlier factor

The main task of outlier detection is to detect data objects which have a different mechanism from the conventional data set. The existing outlier detection methods are mainly divided into two directions: local outliers and global outliers. Aiming at the limitations of the existing outlier detection methods, we propose a novel outlier detection algorithm which is named as kNN-LOF. First, the k-nearest neighbors algorithm is applied to divide different areas for outlier attributes, which is more suitable for outlier detection in different density distributions. Secondly, a hierarchical adjacency order is proposed to hierarchize the neighborhood range according to the link distance. The average sequence distance is calculated from the data objects in the hierarchy, and the reachable distance of an object is redefined to introduce a new local outlier factor. Experimental results show that the proposed algorithm has good performance in improving the accuracy of outlier detection.

An Effective Ensemble-based Framework for Outlier Detection in Evolving Data Streams

In the last few years, data streams have drawn lots of researchers’ attention due to their various applications, such as healthcare monitoring systems, fraud and intrusion detection, the internet of things (IoT), and financial market applications. A data stream is an unbounded sequence of data continually generated over time and is prone to evolution. Outliers in streaming data are the elements that significantly deviate from the majority of elements and then have to be detected as they may be error values or events of interest. Detection of outliers is a challenging issue in streaming data and is one of the most crucial tasks in data stream mining. Existing outlier detection methods for static data are unsuitable for use in data stream settings due to the unique characteristics of streaming data such as unpredictability, uncertainty, high-dimensionality, and changes in data distribution. Thus, in this paper, a novel ensemble learning framework called Ensemble-based Streaming Outlier Detection (ESOD) is presented to perfectly detect outliers over streaming data using a sliding window technique that is updated in response to the incoming events from the data streaming environment to overcome the concept evolution nature of streaming data. The proposed framework has three phases, namely the training phase, testing/offline phase, and outlier detection/online phase. A detection weighted vote technique is used to determine the final decisions for potential outliers. In the extensive experimental study, which was conducted on 11 real-world benchmark datasets, the proposed framework was assessed using many accuracy metrics. The experiment results showed that the proposed framework beats many other state-of-the-art methods.

A fuzzy proximity relation approach for outlier detection in the mixed dataset by using rough entropy-based weighted density method

Data mining is an emerging technology where researchers explore innovative ideas in different domains, particularly detecting anomalies. Instances in the dataset which considerably deviate from others by their common patterns are known as anomalies. The state of being ambiguous and not affording certainty of data exists in this world of nature. Rough Set Theory is a proven methodology which deals with ambiguity and uncertainty of data. Research works that have been done until this point were focused on numeric or categorical type, which fails when the attributes are mixed type. By using fuzzy proximity and ordering relations, the numerical data has been converted to categorical data. This article presented an idea for detecting outliers in mixed data where the weighted density values of attributes and objects are calculated. The proposed approach has been compared with existing outlier detection methods by taking the hiring dataset as an example and benchmarked with Harvard dataverse datasets to prove its efficiency and performance.

Homophily outlier detection in non-IID categorical data

Most of existing outlier detection methods assume that the outlier factors (i.e., outlierness scoring measures) of data entities (e.g., feature values and data objects) are Independent and Identically Distributed (IID). This assumption does not hold in real-world applications where the outlierness of different entities is dependent on each other and/or taken from different probability distributions (non-IID). This may lead to the failure of detecting important outliers that are too subtle to be identified without considering the non-IID nature. The issue is even intensified in more challenging contexts, e.g., high-dimensional data with many noisy features. This work introduces a novel outlier detection framework and its two instances to identify outliers in categorical data by capturing non-IID outlier factors. Our approach first defines and incorporates distribution-sensitive outlier factors and their interdependence into a value-value graph-based representation. It then models an outlierness propagation process in the value graph to learn the outlierness of feature values. The learned value outlierness allows for either direct outlier detection or outlying feature selection. The graph representation and mining approach is employed here to well capture the rich non-IID characteristics. Our empirical results on 15 real-world data sets with different levels of data complexities show that (i) the proposed outlier detection methods significantly outperform five state-of-the-art methods at the 95%/99% confidence level, achieving 10–28% AUC improvement on the 10 most complex data sets; and (ii) the proposed feature selection methods significantly outperform three competing methods in enabling subsequent outlier detection of two different existing detectors.

Data Streams Oriented Outlier Detection Method: A Fast Minimal Infrequent Pattern Mining

Outlier detection is a common method for analyzing data streams. In the existing outlier detection methods, most of methods compute distance of points to solve certain specific outlier detection problems. However, these methods are computationally expensive and cannot process data streams quickly. The outlier detection method based on pattern mining resolves the aforementioned issues, but the existing methods are inefficient and cannot meet requirements of quickly mining data streams. In order to improve the efficiency of the method, a new outlier detection method is proposed in this paper. First, a fast minimal infrequent pattern mining method is proposed to mine the minimal infrequent pattern from data streams. Second, an efficient outlier detection algorithm based on minimal infrequent pattern is proposed for detecting the outliers in the data streams by mining minimal infrequent pattern. The algorithm proposed in this paper is demonstrated by real telemetry data of a satellite in orbit. The experimental results show that the proposed method not only can be applied to satellite outlier detection, but also is superior to the existing methods.

Multi-Level Clustering-Based Outlier’s Detection (MCOD) Using Self-Organizing Maps

Outlier detection is critical in many business applications, as it recognizes unusual behaviours to prevent losses and optimize revenue. For example, illegitimate online transactions can be detected based on its pattern with outlier detection. The performance of existing outlier detection methods is limited by the pattern/behaviour of the dataset; these methods may not perform well without prior knowledge of the dataset. This paper proposes a multi-level outlier detection algorithm (MCOD) that uses multi-level unsupervised learning to cluster the data and discover outliers. The proposed detection method is tested on datasets in different fields with different sizes and dimensions. Experimental analysis has shown that the proposed MCOD algorithm has the ability to improving the outlier detection rate, as compared to the traditional anomaly detection methods. Enterprises and organizations can adopt the proposed MCOD algorithm to ensure a sustainable and efficient detection of frauds/outliers to increase profitability (and/or) to enhance business outcomes.

An efficient approach for outlier detection from uncertain data streams based on maximal frequent patterns

Outlier identification is an important technology to improve the credibility of data and aims at detecting patterns that rarely appear and exhibit a significant difference from other data. However, the detection accuracy achieved by the simple deviation factors of existing pattern-based outlier detection methods is not competitive. In addition, given the large scale of uncertain data streams, the efficiency of many pattern-based outlier detection methods is not high because they use a vast number of frequent patterns to conduct the outlier detection. In this paper, to contend with the uncertain data streams, we propose a maximal-frequent-pattern-based outlier detection method, namely, MFP-OD, for identifying the outliers with a lower time cost. For further improving the detection accuracy of existing outlier detection methods, we design three deviation factors to measure the deviation degree of each transaction. The experimental results indicate that the proposed MFP-OD method can quickly and accurately identify the outliers from uncertain data streams.

Detecting EEG outliers for BCI on the Riemannian manifold using spectral clustering.

Automatically detecting and removing Electroencephalogram (EEG) outliers is essential to design robust brain-computer interfaces (BCIs). In this paper, we propose a novel outlier detection method that works on the Riemannian manifold of sample covariance matrices (SCMs). Existing outlier detection methods run the risk of erroneously rejecting some samples as outliers, even if there is no outlier, due to the detection being based on a reference matrix and a threshold. To address this limitation, our method, Riemannian Spectral Clustering (RiSC), detects outliers by clustering SCMs into non-outliers and outliers, based on a proposed similarity measure. This considers the Riemannian geometry of the space and magnifies the similarity within the non-outlier cluster and weakens it between non-outlier and outlier clusters, instead of setting a threshold. To assess RiSC performance, we generated artificial EEG datasets contaminated by different outlier strengths and numbers. Comparing Hit-False (HF) difference between RiSC and existing outlier detection methods confirmed that RiSC could detect outliers significantly better (p < 0.001). In particular, RiSC improved HF difference the most for datasets with the most severe outlier contamination.

FilterK: A new outlier detection method for k-means clustering of physical activity.

In this paper, a new algorithm denoted as FilterK is proposed for improving the purity of k-means derived physical activity clusters by reducing outlier influence. We applied it to physical activity data obtained with body-worn accelerometers and clustered using k-means. We compared its performance with three existing outlier detection methods: Local Outlier Factor, Isolation Forests and KNN using the ground truth (class labels), average cluster and event purity (ACEP). FilterK provided comparable gains in ACEP (0.581→0.596 compared to 0.580-0.617) whilst removing a lower number of outliers than the other methods (4% total dataset size vs 10% to achieve this ACEP). The main focus of our new outlier detection method is to improve the cluster purities of physical activity accelerometer data, but we also suggest it may be potentially applied to other types of dataset captured by k-means clustering. We demonstrate our method using a k-means model trained on two independent accelerometer datasets (training n=90) and re-applied to an independent dataset (test n=41). Labelled physical activities include lying down, sitting, standing, household chores, walking (laboratory and non-laboratory based), stairs and running. This type of clustering algorithm could be used to assist with identifying optimal physical activity patterns for health.

Embedding-Based Complex Feature Value Coupling Learning for Detecting Outliers in Non-IID Categorical Data

Non-IID categorical data is ubiquitous and common in realworld applications. Learning various kinds of couplings has been proved to be a reliable measure when detecting outliers in such non-IID data. However, it is a critical yet challenging problem to model, represent, and utilise high-order complex value couplings. Existing outlier detection methods normally only focus on pairwise primary value couplings and fail to uncover real relations that hide in complex couplings, resulting in suboptimal and unstable performance. This paper introduces a novel unsupervised embedding-based complex value coupling learning framework EMAC and its instance SCAN to address these issues. SCAN first models primary value couplings. Then, coupling bias is defined to capture complex value couplings with different granularities and highlight the essence of outliers. An embedding method is performed on the value network constructed via biased value couplings, which further learns high-order complex value couplings and embeds these couplings into a value representation matrix. Bidirectional selective value coupling learning is proposed to show how to estimate value and object outlierness through value couplings. Substantial experiments show that SCAN (i) significantly outperforms five state-of-the-art outlier detection methods on thirteen real-world datasets; and (ii) has much better resilience to noise than its competitors.

An Isolation-Based Distributed Outlier Detection Framework Using Nearest Neighbor Ensembles for Wireless Sensor Networks

In recent years, wireless sensor networks have been extensively deployed to collect various data. Due to the effect of harsh environments and the limitation of the computing and communication capabilities of sensor nodes, the quality and reliability of sensor data are affected by outliers. Thus, an effective outlier detection method is essential. The existing outlier detection methods have some drawbacks, such as extra resource consumption introduced by the size growth of a local detector, poor performance of combination methods of local detectors, and the weak adaptability of the dynamic changes of the environment, etc. We propose an isolation-based distributed outlier detection framework using nearest-neighbor ensembles (iNNE) to effectively detect outliers in wireless sensor networks. In our proposed framework, local detectors are constructed in each node by the iNNE algorithm. A new combination method taking advantage of the spatial correlation among sensor nodes for local detectors is presented. The method is based on the weighted voting idea. In addition, we introduce a sliding window to update local detectors, which enables the adaption of dynamic changes in the environment. The extensive experiments are conducted on two classic real sensor datasets. The experimental results show our framework significantly improves the detection accuracy and reduces the false alarm rate compared with other outlier detection frameworks.

A graph-based approach for detecting spatial cross-outliers from two types of spatial point events

Spatial point events are a series of point entities with location information (e.g., longitude and latitude) that describe geographical events, such as crime events. The detection of outliers from spatial point events is very helpful in uncovering unusual geographical phenomena. Existing outlier detection methods mainly focus on single type of events. In practice, it is common that two (or more) types of geographical events can co-occur within a certain spatial region. In this case, the concept of spatial cross-outliers is defined that considers different types of events simultaneously. This study presents an adaptive graph-based approach to fully and accurately detect spatial cross-outliers from two types of spatial point events, which are categorized into target and reference points. First, the cross K-function is utilized to determine whether the reference points are positively dependent on target points or not. On this basis, the spatial cross-neighbourhood relationships between target and reference points are constructed by a two-level edge length constrained Delaunay triangulation and used to quantify the positive dependency degree of reference points on each target point. By considering the spatial distances and local differences of positive dependency degree with respect to target points, the multilevel constrained Delaunay triangulation is further employed to separate spatial cross-outliers. Experiments using both simulated and real-life datasets illustrate that the proposed method can detect spatial cross-outliers in the form of both individual points and collective points with high accuracy and efficiency. Moreover, there is no need to input any parameters.

Evolutionary multi-objective optimization based ensemble autoencoders for image outlier detection

Abstract Image outlier detection has been an important research issue for many computer vision tasks. However, most existing outlier detection methods fail in the high-dimensional image datasets. In order to address this problem, we propose a novel image outlier detection method by combining autoencoder with Adaboost (ADAE). By ensembling many weak autoencoders, our method can better capture the statistical correlations among the features of normal data than the single autoencoder. Therefore, the proposed ADAE is able to determine the outliers efficiently. In order to reduce the many parameters in ADAE, we introduce the Sparse Group Lasso (SGL) constraint into the learning objective of ADAE. We combine Adagrad with Proximal Gradient Descent to optimize this additional learning objective. We also propose the multi-objective evolutionary algorithm to determine the best penalty factors of SGL. By evaluating on several famous image datasets, the detection results testify to the outstanding outlier detection performance of ADAE. The evaluation results also show SGL can make the detection model more compact while maintaining the similar detection performance.

Thresholding-based outlier detection for high-dimensional data

ABSTRACTTraditional outlier detection methods such as the minimum volume ellipsoid method and MCD method are all based on normal distance. As is well known, the norm-based distance is only effective in detecting difference with dense signals. However, these existing approaches will encounter detection power loss under the sparse signals settings. In this paper, we try to solve the problem of detecting outliers under the sparsity assumption by adapting the thresholding method. The proposed outlier detection procedure finds the clean set by the minimum diagonal product algorithm, then the maximum thresholding statistic is employed to identify the outlier. The finite sample performance of our method is evaluated through simulations. Compared with the existing outlier detection methods, simulation results show that our proposed outlier detection procedure is very efficient under sparse settings.

Support high-order tensor data description for outlier detection in high-dimensional big sensor data

The various high-dimensional sensor data can be collected by wireless sensor networks, video monitoring systems and multimedia sensor networks, while High-dimensional sensor data is inherently large-scale because each sensor node has spatial attributes and may also be associated with large amounts of measurement data evolving over time. Detecting outlier in high-dimensional big sensor data is a challenging task. Most of existing outlier detection methods is based on vector representation. However, high-dimensional sensor data is naturally described by tensor representations. The vector-based methods can lead to destroy original structural information and correlation for high-dimensional sensors data, result in the problem of curse of dimensionality, and some outliers cannot be detected. To solve this problem, support high-order tensor data description (STDD) and kernel support high-order tensor data description (KSTDD) are proposed to detect outliers for tensor data. STDD and KSTDD extend support vector data description from vector space to tensor space. KSTDD maintains the structural information of data, avoids the problem caused by the vectorization of tensor data, and improves the performance of outlier detection. Experiments on four sensor datasets show that the proposed method is superior to the traditional vectorized data analysis method.

Speeding problem detection in business surveys: benefits of statistical outlier detection methods

Speeding describes the unusually fast responses provided to survey questions. A characteristic of speeders is that answers by-pass cognitive process. Consequently, this low respondent engagement results in the poor quality and validity of data. The issue at hand is how to detect speeders in a survey. The presumption is the use of different statistical outlier detection methods. This paper presents graphical methods for outlier detection, such as: dot-plot diagrams, scatter diagrams, histograms and box-plot diagrams. Furthermore, the quantitative methods for outlier detection in this paper are the z-score, modified z-score, Dixons’ test, Grubbs’ test, Tietjen-Moore test, Rosners’ or the generalized extreme studentized deviate (ESD) test. The performance of these outlier detection methods was observed on completion times of 217 surveys from enterprises which participated in a web survey on the use of statistical methods, and which use them in their business processes. The analysis has shown that none of the observed outlier detection methods were able to detect speeders in an appropriate and satisfactory way as shown by the threshold. The main reasons for this can be found in slowers, the violations of assumptions on normal distribution and in masking. Hence, existing outlier detection methods should be improved and adjusted in future research in order to detect speeders. The introduction of novel speeders detection methods would be a good choice for future research.

OUP accepted manuscript

SummaryWe propose a procedure based on a high-breakdown mean function estimator to detect outliers in functional data. The robust estimator is obtained from a clean subset of observations, excluding potential outliers, by minimizing the least-trimmed-squares projection coefficients after functional principal component analysis. A threshold rule based on the asymptotic distribution of the functional score-based distance robustly controls the false positive rate and detects outliers effectively. Further improvement in power can be achieved by adding a one-step reweighting procedure. The finite-sample performance of our method demonstrates satisfactory false positive and false negative rates compared with existing outlier detection methods for functional data.

Outlier detection and robust regression for correlated data

Outlier detection has attracted considerable interest in various areas. Existing outlier detection methods usually assume independence of the modeling errors among the data points but this assumption does not hold in a number of applications. In this paper we propose a probabilistic method for outlier detection and robust updating of linear regression problems involving correlated data. First, suspicious data points will be identified using the minimum volume ellipsoid method and the maximum trimmed likelihood method. Then, the outlierness of each suspicious data point will be determined according to the proposed outlier probability in consideration of possible correlation among the data points. The proposed method is assessed and validated through simulated and real data.