Outlier Detection Method Research Articles

ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions

Outlier detection refers to the identification of data points that deviate from a general data distribution. Existing unsupervised approaches often suffer from high computational cost, complex hyperparameter tuning, and limited interpretability, especially when working with large, high-dimensional datasets. To address these issues, we present a simple yet effective algorithm called ECOD (Empirical-Cumulative-distribution-based Outlier Detection), which is inspired by the fact that outliers are often the "rare events" that appear in the tails of a distribution. In a nutshell, ECOD first estimates the underlying distribution of the input data in a nonparametric fashion by computing the empirical cumulative distribution per dimension of the data. ECOD then uses these empirical distributions to estimate tail probabilities per dimension for each data point. Finally, ECOD computes an outlier score of each data point by aggregating estimated tail probabilities across dimensions. Our contributions are as follows: (1) we propose a novel outlier detection method called ECOD, which is both parameter-free and easy to interpret; (2) we perform extensive experiments on 30 benchmark datasets, where we find that ECOD outperforms 11 state-of-the-art baselines in terms of accuracy, efficiency, and scalability; and (3) we release an easy-to-use and scalable (with distributed support) Python implementation for accessibility and reproducibility.

Enhancing anomaly detectors with LatentOut

LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} is a recently introduced algorithm for unsupervised anomaly detection which enhances latent space-based neural methods, namely (Variational) Autoencoders, GANomaly and ANOGan architectures. The main idea behind it is to exploit both the latent space and the baseline score of these architectures in order to provide a refined anomaly score performing density estimation in the augmented latent-space/baseline-score feature space. In this paper we investigate the performance of LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} acting as a one-class classifier and we experiment the combination of LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} with GAAL architectures, a novel type of Generative Adversarial Networks for unsupervised anomaly detection. Moreover, we show that the feature space induced by LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} has the characteristic to enhance the separation between normal and anomalous data. Indeed, we prove that standard data mining outlier detection methods perform better when applied on this novel augmented latent space rather than on the original data space.

Using eigenvalues of distance matrices for outlier detection

Distance or dissimilarity matrices are widely used in applications. We study the relationships between the eigenvalues of the distance matrices and outliers and show that outliers affect the pairwise distances and inflate the eigenvalues. We obtain the eigenvalues of a distance matrix that is affected by k outliers and compare them to the eigenvalues of a distance matrix with a constant structure. We show a discrepancy in the sizes of the eigenvalues of a distance matrix that is contaminated with outliers, present an algorithm and offer a new outlier detection method based on the eigenvalues of the distance matrix. We compare the new distance-based outlier technique with several existing methods under five distributions. The methods are applied to a study of public utility companies and gene expression data.

Outlier detection for partially labeled categorical data based on conditional information entropy

Labeling a large amount of data is exceptionally costly and practically infeasible, and thus available data may have missing labels. In this article, we investigate outlier detection for partially labeled categorical data based on conditional information entropy. Firstly, the equivalence class in a partially labeled categorical decision information system (p-CDIS) is introduced, so that the missing labels can be predicted by use of conditional probability. Then, conditional information entropy in a p-CDIS is calculated, which provides a more comprehensive measure of uncertainty. Additionally, the relative information entropy and relative cardinality in a p-CDIS are proposed. Next, the degree of outlierness and the weight function are presented to find outlier factors. Finally, an outlier detection method in a p-CDIS based on conditional information entropy is proposed, and a corresponding conditional information entropy algorithm (CEOF) is designed. To evaluate the stability of the CEOF algorithm, experiments are performed on ten UCI Machine Learning Repository datasets. Compared with five other algorithms, the proposed method is shown to have good effectiveness and adaptability for categorical data.

Comparison of Classification Models for Breast Cancer Disease Using Multivariate Analysis and Data Mining Approaches

Compared to other cancer types, breast cancer is one of the main causes of death in women. Early cancer detection can significantly increase survival and quality of life. A variety of machine learning prediction algorithms with combination of feature selection approaches have shown to be useful in the detection of breast cancer disease. However, it was discovered that there are still problems with classification accuracy. An outlier-related factor was known to have potential effect on classification accuracy. In order to further improve the classification’s accuracy, the Kmeans approach was used to detect outliers. The major goal of this study was to examine the classification performance of breast cancer disease when feature selection methods were used in combination with K-Means. For experimental purpose, the Coimbra dataset for breast cancer consisting of 116 instances and 10 attributes was used in this study. Multivariate techniques including Principal Component Analysis (PCA), Kernel Principal Component Analysis (KPCA), and Discriminant Analysis (DA) were applied to reduce data dimensions. Meanwhile, four data mining approaches consisting of Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), and Logistic Regression (LR) were compared for classification purpose. The performance measurement was then evaluated using accuracy, precision, specificity, and sensitivity criteria. The results revealed that five combinations approaches (PCA-DT, PCA-RF, KPCA-DT, KPCA-RF, DA-RF) performed better across all four criteria after combining with KMeans technique. Among five combined methods, KPCA with DT outperformed other combination methods with the highest value across precision (76.47 percent) and specificity (71.43 percent). This study suggests the incorporation of feature selection method together with outlier detection method has proved to be more efficient and beneficial for breast cancer classification.

A KRR-UKF robust state estimation method for distribution networks

State estimation is an integral component of energy management systems. Employing a state estimation methodology that is both accurate and resilient is essential for facilitating informed decision-making processes. However, the complex scenarios (unknown noise, low data redundancy, and reconfiguration) of the distribution network pose new challenges for state estimation. In the context of this study, we introduce a state estimation technique known as the kernel ridge regression and unscented Kalman filter. In normal conditions, the non-linear correlation among data and unknown noise increases the difficulty of modeling the distribution network. Thence, kernel ridge regression is developed to map the data into high-dimensional space that transforms the non-linear problem into linear formulations base on the data rather the complicate grid model, which improves model generalization performance and filters out unknown noises. In addition, with the unique prediction correction mechanism of the Kalman method, the kernel ridge regression-mapped state value can be revised by the measurement, which further enhances model accuracy and robustness. During abnormal operating conditions and taking into account the presence of faulty data within the measurement system, we initiate the use of a long short-term memory network and combined convolutional neural network (CNN) model, referred to as the ATT-CNN-GRU. This model is utilized for the prediction of pseudo-measurements. Subsequently, we use an outlier detection method known as ordering points to identify the clustering structure to effectively identify and substitute erroneous data points. Cases on the IEEE-33 bus system and 109-bus system from a city in China show that the method has superior accuracy and robustness.

Robust Fusion of Multi-Source Images for Accurate 3D Reconstruction of Complex Urban Scenes

Integrated reconstruction is crucial for 3D modeling urban scenes using multi-source images. However, large viewpoint and illumination variations pose challenges to existing solutions. A novel approach for accurate 3D reconstruction of complex urban scenes based on robust fusion of multi-source images is proposed. Firstly, georeferenced sparse models are reconstructed from the terrestrial and aerial images using GNSS-aided incremental SfM, respectively. Then, cross-platform match pairs are selected based on point-on-image observability. The terrestrial and aerial images are robustly matched based on the selected match pairs to generate cross-platform tie points. Thirdly, the tie points are triangulated to derive cross-platform 3D correspondences. The 3D correspondences are refined using a novel outlier detection method. Finally, the terrestrial and aerial sparse models are merged based on the refined correspondences, and the integrated model is globally optimized to obtain an accurate reconstruction of the scene. The proposed methodology is evaluated on five benchmark datasets, and extensive experiments are performed. The proposed pipeline is compared with a state-of-the-art methodology and three widely used software packages. Experimental results demonstrate that the proposed methodology outperforms the other pipelines in terms of robustness and accuracy.

Fusing multi-scale fuzzy information to detect outliers

Outlier detection aims to find objects that behave differently from the majority of the data. Existing unsupervised approaches often process data with a single scale, which may not capture the multi-scale nature of the data. In this paper, we propose a novel information fusion model based on multi-scale fuzzy granules and an unsupervised outlier detection algorithm with the fuzzy rough set theory. First, a multi-scale information fusion model is formulated based on fuzzy granules. Then we employ fuzzy approximations to define the outlier factor of multi-scale fuzzy granules centered at each data point. Finally, the outlier score is calculated by aggregating the outlier factors of a set of multi-scale fuzzy granules. Experimental results demonstrate that the proposed method is comparable with or better than the leading outlier detection methods. The codes and datasets are publicly available online at https://github.com/ChenBaiyang/MFIOD.

Neighborhood Rough Residual Network–Based Outlier Detection Method in IoT-Enabled Maritime Transportation Systems

Outlier detection can identify anomalies in large-scale data. To provide reliability and security for Internet of Things (IoT)-enabled maritime transportation systems (MTSs), in this paper we propose an outlier detection method based on the neighborhood rough residual network (NRRN). We calculate the neighborhood approximation accuracy and neighborhood conditional entropy to obtain the neighborhood combined entropy describing the discrimination ability of the condition attribute subset to the information system. We then delete the redundant attributes according to the attribute combination importance derived from the neighborhood combined entropy. The data after attribute reduction are used to train the convolutional neural network, and the residual network (ResNet50) is used to avoid the degradation of model performance caused by the increase in the number of network layers. The proposed method is compared with mainstream outlier detection algorithms on a fishing vessel operation dataset. Experiments show that the proposed method can greatly improve the accuracy of outlier detection while taking into account interpretability and computational efficiency, thereby ensuring the data integrity of IoT-enabled MTSs.

Detecting unexpected growths in health technologies expenditures: the case of MIPRES in Colombia

We developed an algorithm to explore unexpected growth in the usage and costs of health technologies. We exploit data from the expenditures on technologies funded by the Colombian government under the compulsory insurance system, where all prescriptions for technologies not included in an explicit list must be registered in a centralized information system, covering the period from 2017 to 2022. The algorithm consists of two steps: an outlier detection method based on the density of the expenditures for selecting a first set of technologies to consider (39 technologies out of 106,957), and two anomaly detection models for time series to determine which insurance companies, health providers, and regions have the most notorious increases. We have found that most medicines associated with atypical behavior and significant monetary growth could be linked to the use of recently introduced drugs in the market. These drugs have valid patents and very specific clinical indications, often involving high-cost pharmacological treatments. The most relevant case is the Burosumab, approved in 2018 to treat a rare genetic disorder affecting skeletal growth. Secondly, there is clear evidence of anomalous increasing trend evolutions in the identified enteral nutritional support supplements or Food for Special Medical Purposes. The health system did not purchase these products before July 2021, but in 2022 they represented more than 500,000 USD per month.

A data-adaptive method for outlier detection from functional data

A data-adaptive method for outlier detection from functional data

New approaches and technical considerations in detecting outlier measurements and trajectories in longitudinal children growth data

BackgroundGrowth studies rely on longitudinal measurements, typically represented as trajectories. However, anthropometry is prone to errors that can generate outliers. While various methods are available for detecting outlier measurements, a gold standard has yet to be identified, and there is no established method for outlying trajectories. Thus, outlier types and their effects on growth pattern detection still need to be investigated. This work aimed to assess the performance of six methods at detecting different types of outliers, propose two novel methods for outlier trajectory detection and evaluate how outliers affect growth pattern detection.MethodsWe included 393 healthy infants from The Applied Research Group for Kids (TARGet Kids!) cohort and 1651 children with severe malnutrition from the co-trimoxazole prophylaxis clinical trial. We injected outliers of three types and six intensities and applied four outlier detection methods for measurements (model-based and World Health Organization cut-offs-based) and two for trajectories. We also assessed growth pattern detection before and after outlier injection using time series clustering and latent class mixed models. Error type, intensity, and population affected method performance.ResultsModel-based outlier detection methods performed best for measurements with precision between 5.72-99.89%, especially for low and moderate error intensities. The clustering-based outlier trajectory method had high precision of 14.93-99.12%. Combining methods improved the detection rate to 21.82% in outlier measurements. Finally, when comparing growth groups with and without outliers, the outliers were shown to alter group membership by 57.9 -79.04%.ConclusionsWorld Health Organization cut-off-based techniques were shown to perform well in few very particular cases (extreme errors of high intensity), while model-based techniques performed well, especially for moderate errors of low intensity. Clustering-based outlier trajectory detection performed exceptionally well across all types and intensities of errors, indicating a potential strategic change in how outliers in growth data are viewed. Finally, the importance of detecting outliers was shown, given its impact on children growth studies, as demonstrated by comparing results of growth group detection.

Outlier detection using iterative adaptive mini-minimum spanning tree generation with applications on medical data.

As an important technique for data pre-processing, outlier detection plays a crucial role in various real applications and has gained substantial attention, especially in medical fields. Despite the importance of outlier detection, many existing methods are vulnerable to the distribution of outliers and require prior knowledge, such as the outlier proportion. To address this problem to some extent, this article proposes an adaptive mini-minimum spanning tree-based outlier detection (MMOD) method, which utilizes a novel distance measure by scaling the Euclidean distance. For datasets containing different densities and taking on different shapes, our method can identify outliers without prior knowledge of outlier percentages. The results on both real-world medical data corpora and intuitive synthetic datasets demonstrate the effectiveness of the proposed method compared to state-of-the-art methods.

Outlier detection for incomplete real-valued data via rough set theory and granular computing

Outlier detection is an important topic in data mining. An information system (IS) is a database that shows relationships between objects and attributes. A real-valued information system (RVIS) is an IS whose information values are real numbers. People often encounter missing values during data processing. A RVIS with the miss values is an incomplete real-valued information system (IRVIS). Due to the presence of the missing values, the distance between two information values is difficult to determine, so the existing outlier detection rarely considered an IS with the miss values. This paper investigates outlier detection for an IRVIS via rough set theory and granular computing. Firstly, the distance between two information values on each attribute of an IRVIS is introduced, and the parameter λ to control the distance is given. Then, the tolerance relation on the object set is defined according to the distance, and the tolerance class is obtained, which is regarded as an information granule. After then, λ-lower and λ-upper approximations in an IRVIS are put forward. Next, the outlier factor of every object in an IRVIS is presented. Finally, outlier detection method for IRVIS via rough set theory and granular computing is proposed, and the corresponding algorithms is designed. Through the experiments, the proposed method is compared with other methods. The experimental results show that the designed algorithm is more effective than some existing algorithms in an IRVIS. It is worth mentioning that for comprehensive comparison, ROC curve and AUC value are used to illustrate the advantages of the proposed method.

Human Activity Recognition with Smartwatch Data by using Mahalanobis Distance-Based Outlier Detection and Ensemble Learning Methods

Recognition of human activities is part of smart healthcare applications. In this context, the detection of human actions with high accuracy has been a field that has been working for many years. With the increase in the usage of smart devices, smartphones and smartwatches have become the constant equipment of these studies thanks to their internal sensors. Sometimes abnormal data are included in data sets due to the way the data were collected and for reasons arising from the sensors. For this reason, it becomes important to detect outlier data. In this study, step counter and heart rate sensors were used in addition to an accelerometer and gyroscope in order to detect human activities. Afterward, the outliers were detected and cleared with a Mahalanobis distance-based approach. With the aim of achieving a better classification performance, machine learning methods were used by strengthening them with ensemble learning methods. The obtained results showed that step counter, heart rate sensors, and ensemble learning methods positively affect the success of the classification. In addition, it was found that the Mahalanobis distance-based outlier detection method increased the classification accuracy significantly.

A novel subspace outlier detection method by entropy-based clustering algorithm

Subspace outlier detection has emerged as a practical approach for outlier detection. Classical full space outlier detection methods become ineffective in high dimensional data due to the “curse of dimensionality”. Subspace outlier detection methods have great potential to overcome the problem. However, the challenge becomes how to determine which subspaces to be used for outlier detection among a huge number of all subspaces. In this paper, firstly, we propose an intuitive definition of outliers in subspaces. We study the desirable properties of subspaces for outlier detection and investigate the metrics for those properties. Then, a novel subspace outlier detection algorithm with a statistical foundation is proposed. Our method selectively leverages a limited set of the most interesting subspaces for outlier detection. Through experimental validation, we demonstrate that identifying outliers within this reduced set of highly interesting subspaces yields significantly higher accuracy compared to analyzing the entire feature space. We show by experiments that the proposed method outperforms competing subspace outlier detection approaches on real world data sets.

A rapid outliers detection and correction method for external ballistic velocity measurement data

Due to the influence of various sudden and abnormal factors during the tracking environment, equipment working conditions, and operation process, there are inevitable outliers in the tracking measurement process of aircraft such as carrier rockets, artificial satellites, and missiles. Therefore, the prerequisite for ensuring the reliability of processing results is to timely and accurately detect and correct outliers. In this paper, we proposed the sliding window-based variable degree B-spline function method (SWVD B-spline), which can handle isolated outliers and spotted outliers. SWVD B-spline uses variable degree B-spline function to model observed data in sliding windows, which can detect and correct outliers point by point along with window sliding. Then, we propose an initial window data selection method to remove outliers in initial windows to ensure the processing effect. In addition, because there are often inflection points in external ballistic velocity measurement data, differential evolution is used to optimize variable degree B-spline in windows that include inflection points to improve processing accuracy. The experimental results verify that SWVD B-spline can handle various outliers rapidly and efficiently.

Inclusion of Control Data in Fits to Concentration-Response Curves Improves Estimates of Half-Maximal Concentrations.

Concentration-response curves, in which the effect of varying the concentration on the response of an assay is measured, are widely used to evaluate biological effects of chemical compounds. While National Center for Advancing Translational Sciences guidelines specify that readouts should be normalized by the controls, recommended statistical analyses do not explicitly fit to the control data. Here, we introduce a nonlinear regression procedure based on maximum likelihood estimation that determines parameters for the classical Hill equation by fitting the model to both the curve and the control data. Simulations show that the proposed procedure provides more precise parameters compared with previously prescribed practices. Analysis of enzymatic inhibition data from the COVID Moonshot demonstrates that the proposed procedure yields a lower asymptotic standard error for estimated parameters. Benefits are most evident in the analysis of the incomplete curves. We also find that Lenth's outlier detection method appears to determine parameters more precisely.

Detection of outliers with respect to a MUSIC geotechnical database

This study proposes a novel method that addresses a nontraditional class of outlier detection problems. The purpose of most outlier detection methods in the literature is to detect outliers within a dataset. A record can be considered an outlier if it is distinct from the regular records in the dataset. However, the purpose of the novel outlier detection method proposed in this study is to detect outlier data groups (a data group may denote a site or a project) with respect to a soil/rock property "MUSIC" database. A data group is an outlier group if its characteristics (mean, variance, correlation, or higher order dependency) are distinct from the regular data groups in the database. This study frames the outlier detection problem into a formal hypothesis testing problems with the null hypothesis that “the target data group is identically distributed as the regular groups in the database.” With the hierarchical Bayesian model previously developed by the first two authors, the p-value for this hypothesis testing problem can be estimated rigorously. Numerical and real examples show that the p-value can effectively detect outlier data groups as well as outlier records with respect to a database.

Multi-view Outlier Detection via Graphs Denoising

Recently, multi-view outlier detection attracts increasingly more attention. Although existing multi-view outlier detection methods have demonstrated promising performance, they still suffer from some problems. Firstly, many methods make the assumption that the data have a clear clustering structure and detect the outliers by using some off-the-shelf clustering methods. Therefore, the performance of these methods depends on the clustering methods they used, and thus these methods are hard to handle complicated data. Secondly, some methods ignore the complicated structure or distribution of class outliers and directly learn a consensus representation by simply combining the representation of different views linearly. To tackle these problems, we propose a novel method named Multi-view Outlier Detection with Graph Denoising (MODGD). We first construct a graph for each view, and then learn a consensus graph by ensembling the multiple graphs. When fusing the multiple graphs, we explicitly characterize and extract the structured outliers on each graph and recover the multiple clean graphs for the ensemble. During the process of multiple graph denoising and fusion, we carefully design an outlier measurement criterion based on the characteristics of attribute and class outliers. The extensive experiments on benchmark data sets demonstrate the effectiveness and superiority of the proposed method. The codes of this paper are released in http://Doctor-Nobody.github.io/codes/MODGD.zip.