High-dimensional Outlier Detection Research Articles

A High-dimensional Outlier Detection Approach Based on Local Coulomb Force

Traditional outlier detections are inadequate for high-dimensional data analysis due to the interference of distance tending to be concentrated (curse of dimensionality). Inspired by the Coulombs law, we propose a new high-dimensional data similarity measure vector, which consists of outlier Coulomb force and outlier Coulomb resultant force. Outlier Coulomb force not only effectively gauges similarity measures among data objects, but also fully reflects differences among dimensions of data objects by vector projection in each dimension. More importantly, Coulomb resultant force can effectively measure deviations of data objects from a data center, making detection results interpretable. We introduce a new neighborhood outlier factor, which drives the development of a high-dimensional outlier detection algorithm. In our approach, attribute values with a high deviation degree is treated as interpretable information of outlier data. Finally, we implement and evaluate our algorithm using the UCI and synthetic datasets. Our experimental results show that the algorithm effectively alleviates the interference of Curse of Dimensionality. The findings confirm that high-dimensional outlier data originated by the algorithm are interpretable.

High-dimensional outlier detection using random projections

There exist multiple methods to detect outliers in multivariate data in the literature, but most of them require to estimate the covariance matrix. The higher the dimension, the more complex the estimation of the matrix becoming impossible in high dimensions. In order to avoid estimating this matrix, we propose a novel random projection-based procedure to detect outliers in Gaussian multivariate data. It consists in projecting the data in several one-dimensional subspaces where an appropriate univariate outlier detection method, similar to Tukey’s method but with a threshold depending on the initial dimension and the sample size, is applied. The required number of projections is determined using sequential analysis. Simulated and real datasets illustrate the performance of the proposed method.

A new density-based subspace selection method using mutual information for high dimensional outlier detection

Outlier detection in high dimensional data faces the challenge of curse of dimensionality, where irrelevant features may prevent detection of outliers. In this research, we propose a novel efficient unsupervised density-based subspace selection for outlier detection in the projected subspace. First, the Maximum-Relevance-to-Density algorithm(MRD) is proposed to select the relevant subspace based on the mutual information. Then, applying the concept of redundancy among features, we present an efficient relevant subspace selection method called minimum-Redundancy-Maximum-Relevance-to-Density (mRMRD). Finally, the degree of outlierness of data points in the corresponding relevant subspace is computed based on Local Outlier Factor(LOF). Experimental results on both real and synthetic data demonstrate that the proposed algorithms – based on MRD and mRMRD criteria – increase the accuracy of outlier detection while reducing computational complexity and execution time. Moreover, as the dimensionality increases, the accuracy of outlier detection on mRMRD-based relevant subspace is higher than MRD-based relevant subspace. This verifies that the proposed mRMRD-based subspace selection algorithm can efficiently select the subspace by considering the relevance between features.

Subspace rotations for high-dimensional outlier detection

We propose a new two-stage procedure for detecting multiple outliers when the dimension of the data is much larger than the available sample size. In the first stage, the data are split into two disjoint sets, one containing non-outliers and the other containing the rest of the data that are considered as potential outliers. In the second stage, a series of hypothesis tests is conducted to test the abnormality of each potential outlier. A nonparametric test based on uniform random rotations is adopted for hypothesis testing. The power of the proposed test is studied under a high-dimensional asymptotic framework, and its finite-sample exactness is established under mild conditions. Numerical studies based on simulated examples and face recognition data suggest that the proposed approach is superior to the existing methods, especially in terms of false identification of non-outliers.

Local projections for high-dimensional outlier detection

A novel approach for outlier detection is proposed, called local projections, which is based on concepts of the Local Outlier Factor (LOF) (Breunig et al. in Lof: identifying density-based local outliers. In: ACM sigmod record, ACM, volume 29, pp. 93–104, 2000) and ROBPCA (Hubert et al. in Technometrics 47(1):64–79, 2005). By using aspects of both methods, this algorithm is robust towards noise variables and is capable of performing outlier detection in multi-group situations. The idea is to focus on local descriptions of the observations and their neighbors using linear projections. The outlyingness of an observation is determined by a weighted distance of the observation to all identified projection spaces, with weights depending on the appropriateness of the local description. Experiments with simulated and real data demonstrate the usefulness of this method when compared to existing outlier detection algorithms.

Sparse Modeling-Based Sequential Ensemble Learning for Effective Outlier Detection in High-Dimensional Numeric Data

The large proportion of irrelevant or noisy features in real-life high-dimensional data presents a significant challenge to subspace/feature selection-based high-dimensional outlier detection (a.k.a. outlier scoring) methods. These methods often perform the two dependent tasks: relevant feature subset search and outlier scoring independently, consequently retaining features/subspaces irrelevant to the scoring method and downgrading the detection performance. This paper introduces a novel sequential ensemble-based framework SEMSE and its instance CINFO to address this issue. SEMSE learns the sequential ensembles to mutually refine feature selection and outlier scoring by iterative sparse modeling with outlier scores as the pseudo target feature. CINFO instantiates SEMSE by using three successive recurrent components to build such sequential ensembles. Given outlier scores output by an existing outlier scoring method on a feature subset, CINFO first defines a Cantelli's inequality-based outlier thresholding function to select outlier candidates with a false positive upper bound. It then performs lasso-based sparse regression by treating the outlier scores as the target feature and the original features as predictors on the outlier candidate set to obtain a feature subset that is tailored for the outlier scoring method. Our experiments show that two different outlier scoring methods enabled by CINFO (i) perform significantly better on 11 real-life high-dimensional data sets, and (ii) have much better resilience to noisy features, compared to their bare versions and three state-of-the-art competitors. The source code of CINFO is available at https://sites.google.com/site/gspangsite/sourcecode.

Distance-based outlier detection for high dimension, low sample size data

ABSTRACTDespite the popularity of high dimension, low sample size data analysis, there has not been enough attention to the sample integrity issue, in particular, a possibility of outliers in the data. A new outlier detection procedure for data with much larger dimensionality than the sample size is presented. The proposed method is motivated by asymptotic properties of high-dimensional distance measures. Empirical studies suggest that high-dimensional outlier detection is more likely to suffer from a swamping effect rather than a masking effect, thus yields more false positives than false negatives. We compare the proposed approaches with existing methods using simulated data from various population settings. A real data example is presented with a consideration on the implication of found outliers.

Subspace histograms for outlier detection in linear time

Outlier detection algorithms are often computationally intensive because of their need to score each point in the data. Even simple distance-based algorithms have quadratic complexity. High-dimensional outlier detection algorithms such as subspace methods are often even more computationally intensive because of their need to explore different subspaces of the data. In this paper, we propose an exceedingly simple subspace outlier detection algorithm, which can be implemented in a few lines of code, and whose complexity is linear in the size of the data set and the space requirement is constant. We show that this outlier detection algorithm is much faster than both conventional and high-dimensional algorithms and also provides more accurate results. The approach uses randomized hashing to score data points and has a neat subspace interpretation. We provide a visual representation of this interpretability in terms of outlier sensitivity histograms. Furthermore, the approach can be easily generalized to data streams, where it provides an efficient approach to discover outliers in real time. We present experimental results showing the effectiveness of the approach over other state-of-the-art methods.

Multiple outliers detection in sparse high-dimensional regression

ABSTRACTThe presence of outliers would inevitably lead to distorted analysis and inappropriate prediction, especially for multiple outliers in high-dimensional regression, where the high dimensionality of the data might amplify the chance of an observation or multiple observations being outlying. Noting that the detection of outliers is not only necessary but also important in high-dimensional regression analysis, we, in this paper, propose a feasible outlier detection approach in sparse high-dimensional linear regression model. Firstly, we search a clean subset by use of the sure independence screening method and the least trimmed square regression estimates. Then, we define a high-dimensional outlier detection measure and propose a multiple outliers detection approach through multiple testing procedures. In addition, to enhance efficiency, we refine the outlier detection rule after obtaining a relatively reliable non-outlier subset based on the initial detection approach. By comparison studies based on Monte Carlo simulation, it is shown that the proposed method performs well for detecting multiple outliers in sparse high-dimensional linear regression model. We further illustrate the application of the proposed method by empirical analysis of a real-life protein and gene expression data.

A regression sequences based method for high dimensional outlier detection

In recent years, high dimensional space outlier detection has attracted wide attention. High dimensional space outlier detection plays a very important role in many fields such as network intrusion detection, credit card fraud, electronic commerce crime, medical diagnosis and anti-terrorism. The purpose of high-dimensional outlier detection is to find a small number of objects in a high-dimensional dataset, which has a special behavior or abnormal behavior compared with most of the rest of the objects in the high-dimensional dataset. The existing high dimensional space outlier detection method cannot effectively deal with the problem of uncertain and incomplete data. In this paper, the concept of rough set boundary and the high dimensional space outlier detection method based on regression distance proposed by Knorr et al are combined together. We propose a new method for the definition and detection of high dimensional space outliers based on the regression sequence in the framework of rough set. For this method, we design a corresponding high dimensional space outlier detection algorithm (HDSOD) and verify the validity of the algorithm HDSOD by experiments on high dimensional data set. The experimental results show that the proposed method provides a new method to deal with the uncertain and incomplete data in high dimensional space outlier detection.

Visual interactive evolutionary algorithm for high dimensional outlier detection and data clustering problems

Usual visualisation techniques for multidimensional datasets, such as parallel coordinates and scatterplot matrices, do not scale well to high numbers of dimensions. A common approach to solve this problem is dimensionality selection. Existing dimensionality selection techniques usually select pertinent dimension subsets that are significant to the user without loose of information. We present concrete cooperation between automatic algorithms, interactive algorithms and visualisation tools: the evolutionary algorithm is used to obtain optimal dimension subsets which represent the original dataset without loosing information for unsupervised mode (clustering or outlier detection). The last effective cooperation is a visualisation tool used to present the user interactive evolutionary algorithm results and let him actively participate in evolutionary algorithm searching with more efficiency resulting in a faster evolutionary algorithm convergence. We have implemented our approach and applied it to real dataset to confirm this approach is effective for supporting the user in the exploration of high dimensional datasets.

An effective and efficient algorithm for high-dimensional outlier detection

The outlier detection problem has important applications in the field of fraud detection, network robustness analysis, and intrusion detection. Most such applications are most important for high-dimensional domains in which the data can contain hundreds of dimensions. Many recent algorithms have been proposed for outlier detection that use several concepts of proximity in order to find the outliers based on their relationship to the other points in the data. However, in high-dimensional space, the data are sparse and concepts using the notion of proximity fail to retain their effectiveness. In fact, the sparsity of high-dimensional data can be understood in a different way so as to imply that every point is an equally good outlier from the perspective of distance-based definitions. Consequently, for high-dimensional data, the notion of finding meaningful outliers becomes substantially more complex and nonobvious. In this paper, we discuss new techniques for outlier detection that find the outliers by studying the behavior of projections from the data set.