Outlier Detection Method Research Articles

Evaluating methods of outlier detection when benchmarking clinical registry data–a simulation study

Evaluating methods of outlier detection when benchmarking clinical registry data–a simulation study

Prediction of cyanotic and acyanotic congenital heart disease using machine learning models.

Congenital heart disease is most commonly seen in neonates and it is a major cause of pediatric illness and childhood morbidity and mortality. To identify and build the best predictive model for predicting cyanotic and acyanotic congenital heart disease in children during pregnancy and identify their potential risk factors. The data were collected from the Pediatric Cardiology Department at Chaudhry Pervaiz Elahi Institute of Cardiology Multan, Pakistan from December 2017 to October 2019. A sample of 3900 mothers whose children were diagnosed with cyanotic or acyanotic congenital heart disease was taken. Multivariate outlier detection methods were used to identify the potential outliers. Different machine learning models were compared, and the best-fitted model was selected using the area under the curve, sensitivity, and specificity of the models. Out of 3900 patients included, about 69.5% had acyanotic and 30.5% had cyanotic congenital heart disease. Males had more cases of acyanotic (53.6%) and cyanotic (54.5%) congenital heart disease as compared to females. The odds of having cyanotic was 1.28 times higher for children whose mothers used more fast food frequently during pregnancy. The artificial neural network model was selected as the best predictive model with an area under the curve of 0.9012, sensitivity of 65.76%, and specificity of 97.23%. Children having a positive family history are at very high risk of having cyanotic and acyanotic congenital heart disease. Males are more at risk and their mothers need more care, good food, and physical activity during pregnancy. The best-fitted model for predicting cyanotic and acyanotic congenital heart disease is the artificial neural network. The results obtained and the best model identified will be useful for medical practitioners and public health scientists for an informed decision-making process about the earlier diagnosis and improve the health condition of children in Pakistan.

An ORSAC method for data cleaning inspired by RANSAC

In classification problems, mislabeled data can have a dramatic effect on the capability of a trained model. The traditional method of dealing with mislabeled data is through expert review. However, this is not always ideal, due to the large volume of data in many classification datasets, such as image datasets supporting deep learning models, and the limited availability of human experts for reviewing the data. Herein, we propose an ordered sample consensus (ORSAC) method to support data cleaning by flagging mislabeled data. This method is inspired by the random sample consensus (RANSAC) method for outlier detection. In short, the method involves iteratively training and testing a model on different splits of the dataset, recording misclassifications, and flagging data that is frequently misclassified as probably mislabeled. We evaluate the method by purposefully mislabeling subsets of data and assessing the method’s capability to find such data. We demonstrate with three datasets, a mosquito image dataset, CIFAR-10, and CIFAR-100, that this method is reliable in finding mislabeled data with a high degree of accuracy. Our experimental results indicate a high proficiency of our methodology in identifying mislabeled data across these diverse datasets, with performance assessed using different mislabeling frequencies.

Genome‐Wide Diversity in Lowland and Highland Maize Landraces From Southern South America: Population Genetics Insights to Assist Conservation

ABSTRACTMaize (Zea mays ssp. mays L.) landraces are traditional American crops with high genetic variability that conform a source of original alleles for conventional maize breeding. Northern Argentina, one the southernmost regions of traditional maize cultivation in the Americas, harbours around 57 races traditionally grown in two regions with contrasting environmental conditions, namely, the Andean mountains in the Northwest and the tropical grasslands and Atlantic Forest in the Northeast. These races encounter diverse threats to their genetic diversity and persistence in their regions of origin, with climate change standing out as one of the major challenges. In this work, we use genome‐wide SNPs derived from ddRADseq to study the genetic diversity of individuals representing the five groups previously described for this area. This allowed us to distinguish two clearly differentiated gene pools, the highland northwestern maize (HNWA) and the floury northeastern maize (FNEA). Subsequently, we employed essential biodiversity variables at the genetic level, as proposed by the Group on Earth Observations Biodiversity Observation Network (GEO BON), to evaluate the conservation status of these two groups. This assessment encompassed genetic diversity (Pi), inbreeding coefficient (F) and effective population size (Ne). FNEA showed low Ne values and high F values, while HNWA showed low Ne values and low Pi values, indicating that further genetic erosion is imminent for these landraces. Outlier detection methods allowed identification of putative adaptive genomic regions, consistent with previously reported flowering‐time loci and chromosomal regions displaying introgression from the teosinte Zea mays ssp. mexicana. Finally, species distribution models were obtained for two future climate scenarios, showing a notable reduction in the potential planting area of HNWA and a shift in the cultivation areas of FNEA. These results suggest that maize landraces from Northern Argentina may be unable to cope with climate change. Therefore, active conservation policies are advisable.

A probabilistic approach driven credit card anomaly detection with CBLOF and isolation forest models

Businesses throughout the world are starting to accept more and more electronic payment alternatives for in-person and online purchases. Unusual behaviour such as internet fraud and payment defaults, which may lead to significant monetary losses, have become increasingly widespread as credit card use in online buying has expanded. To find a solution to this problem, researchers looked into a number of different machine learning classifiers that may identify irregularities in the data pertaining to credit card transactions. Overlapping class samples and an uneven distribution of classes make it hard to find outliers in this data. General learning algorithms may favour samples from the majority class, and the detection rate of anomalies in samples from minority classes is thus low. Our proposed Credit Card Outlier Detection (CCOD) model incorporates many machine learning algorithms to enhance the detection rates of credit card anomalies. Utilising the stratified sampling technique and the k-fold cross-validation process, we address data imbalance and overfitting. Working with optimized selected features rather than all features reduces the danger of overfitting and improves model performance. It has been noted that the findings of Cluster-Based Local Outlier Factor (CBLOF) classifier on European Credit Card Dataset and Isolation Forest classifier on German credit card dataset were superior to those of the other classifiers with Mathew correlation coefficient value of 0.95 and 0.97 respectively. The CBLOF, an advanced outlier detection method that evaluates the local density of data points within clusters. By comparing the density of a point to the densities of its surrounding clusters, CBLOF identifies outliers based on their deviation from normal cluster behaviour. In general, this CCOD model shows that it is better at dealing with issues like uneven class distribution, overfitting, and classes that overlap.

Outlier Detection in Bayesian Neural Networks

Describing uncertainty is one of the major issues in modern deep learning. Artificial Intelligence models could be used with greater confidence by having solid methods for identifying and quantifying uncertainty. This article proposes two alternative methods for outlier detection in Bayesian Neural Networks used for classification tasks. This is done by looking for unusual pre-activation neuron values in the last layer of a Bayesian Neural Network. The proposed methods are compared to a baseline method for outlier detection, Predictive Entropy, on three datasets: a simulated dataset, the MNIST dataset, and the Breast Cancer Wisconsin dataset. In addition, we introduce a method for separating In-Between and Out-Of-Distribution outliers. The results indicate that the proposed methods' performance depends on the dataset type. In the case of a simple simulated dataset, we see that the proposed methods outperform the baseline method. The proposed method also outperforms the baseline method on the Breast Cancer Wisconsin dataset. However, when tested on the MNIST image dataset, we observed that the baseline is better than the proposed method. As expected, for all three datasets, an OR-combination of the two methods gives slightly better POWER score than the two methods separately, but this comes at the price of also obtaining a higher FDR score.

Committee machine learning: A breakthrough in the precise prediction of CO2 storage mass and oil production volumes in unconventional reservoirs

Accurate prediction of CO2 storage mass and cumulative oil production is critical in the context of combining subsurface carbon capture with enhanced oil recovery (CCS-EOR). This study introduces a novel committee machine-learning Gaussian Process Regression (CML-GPR) model, which integrates three well-established machine-learning algorithms—Random Forest (RF), Multi-Layer Extreme Learning Machine (MELM), and Generalized Regression Neural Network (GRNN). The combination of these models leverages the strengths of each algorithm: RF captures nonlinear relationships, MELM enhances computational efficiency, and GRNN provides smooth, generalized predictions. By integrating these complementary techniques, the CML-GPR model demonstrates significant improvements in predictive accuracy over individual models, addressing limitations in their performance. The model predicts CO2 storage mass and cumulative oil production based on nine key reservoir input variables, including depth, porosity, and CO2 injection rate, among others. Utilizing a large dataset of 21,193 data points from reservoir simulations, a Mahalanobis distance-based outlier detection method further refines the input data quality. The CML-GPR model achieves root mean square error (RMSE) values of 0.49 million metric tons for CO2 storage mass and 13.68 million barrels for cumulative oil production, significantly outperforming individual models. The CML-GPR model provides a robust tool for optimizing CO2 storage capacity and oil recovery, with practical implications for real-world reservoir management, ensuring more efficient and reliable CCS-EOR operations. This study represents a pioneering advancement in predictive modeling, offering valuable insights for optimizing both CO2 storage and enhanced oil recovery in complex reservoirs.

Fast and explainable outlier detection in geotechnical database

ABSTRACT Outlier detection is an important task in data-driven geotechnics. It is noted that many existing methods for outlier detection (e.g. Bayesian learning, neural networks) may pose computational challenge for conventional geotechnical practitioners. Towards this aspect, this study proposes a simple, fast, and explainable method for detecting outliers in geotechnical database. The principle of the method is that for a data point to be labelled as a potential outlier, the probability of observing a data point as extreme as that value should be low. To account for outliers in left-tail and right-tail, the skewness of the dataset is incorporated, and an indicator referred to as outlier score ( > 0 ) is assigned to each data point. The method also provides another indicator (referred to as dimensional outlier score) to identify which dimensions/soil properties contribute to the outlierness; hence, it is explainable. The method doesn't require any time-consuming learning or sampling procedures; hence, it is quite fast and practitioner-friendly. Multiple numerical examples are utilised to demonstrate the capability first. Finally, four publicly available geotechnical databases are utilised to demonstrate the outlier detection task. The results suggest that the outliers identified using the proposed method can be meaningful from a geotechnical point of view.

Traffic Flow Outlier Detection for Smart Mobility Using Gaussian Process Regression Assisted Stochastic Differential Equations

Current methods for detecting outliers in traffic streaming data often struggle to capture real-time dynamic changes in traffic conditions and differentiate between genuine changes and anomalies. This study proposes a novel approach to outlier detection in traffic streaming data that effectively addresses stochasticity and uncertainty in observations. The proposed method utilizes Stochastic Differential Equations (SDEs) and Gaussian Process Regression (GPR). By employing SDEs, we can capture drift and diffusion estimates in traffic streaming data, providing a more comprehensive modeling of the data generation process. Integrating GPR allows precise Bayesian posterior inferences for outlier detection within the SDE framework. To improve practicality, we introduce a flexible threshold-setting mechanism using statistical testing to control the false positive rate. This adaptability helps strike a balance between model fitting and complexity in outlier detection. Compared to traditional SDE-based methods, our SDE-GPR outlier detection method demonstrates enhanced robustness and better adaptability to the complexities of traffic systems. This is evidenced through an empirical study using time series data collected in California, USA. Overall, this study introduces a more advanced and accurate approach to outlier detection in traffic streaming data, paving the way for improved real-time traffic condition monitoring and management.

Expanding the landscape of systemic sclerosis-related autoantibodies through RNA immunoprecipitation coupled with massive parallel sequencing

ObjectivesSystemic sclerosis (SSc)-related autoantibodies are widely used diagnostic and prognostic biomarkers. This study aimed to develop a new assay for detecting anti-ribonucleoprotein autoantibodies in SSc based on RNA immunoprecipitation (RNA IP) coupled with massive parallel sequencing. MethodsSerum samples and clinical data were collected from 307 SSc patients. Among these, 57 samples underwent analysis using a new protocol that combines RNA IP with massive parallel sequencing (RIP-Seq). Filtering strategies and statistical outlier detection methods were applied to select RNA molecules that could represent novel ribonucleoprotein autoantigens associated with SSc. ResultsAmong the 30,966 different RNA molecules identified by RIP-Seq in 57 SSc patients, 197 were ultimately selected. These included all RNA molecules previously identified by RNA IP, which were found to exhibit high counts almost exclusively in samples positive for the autoantibodies associated to the corresponding RNA molecule, indicating high sensitivity and specificity of the RIP-Seq technique. C/D box snoRNAs were the most abundant RNA type identified. The immunoprecipitation patterns of the detected C/D box snoRNAs varied among patients and could be associated with different clinical phenotypes. In addition, other ribonucleoproteins were identified, which could be potential targets for previously undescribed SSc-related autoantibodies. These include H/ACA box snoRNPs, vault complexes, mitochondrial tRNA synthetases, and 7SK snRNP. ConclusionA novel RIP-Seq assay has been developed to detect autoantibodies targeting ribonucleoprotein complexes in SSc patients. This method successfully identified RNA molecules associated with ribonucleoproteins known to be targeted by SSc-related autoantibodies, validating both the assay and the analysis strategy. Additionally, this approach uncovered RNA molecules associated with ribonucleoproteins that were not previously identified as targets of SSc patients’ sera, suggesting potential new autoantibody candidates in this disease.

Partition-based outlier detection for high dimensional data

Abstract Outlier detection has become increasingly important for high dimensional data (HDD) before data analysis. Although the multivariate outlier detection methods (ODMs) have been extensively studied in the research literature, there are few ODMs for ODMs when a correlation exists between data’s components. In this paper, we introduce a partition-based estimator of the minimum covariance determinant for HDD, along with a rapid algorithm designed to identify the clean subset. Additionally, an OD threshold is employed by examining the asymptotic distribution of the outlying measure distance to pinpoint outliers. Moreover, a one-step reweighting scheme is introduced to enhance the detection efficiency of the process. At last, simulation results demonstrate that the proposed RPMCD methods exhibit superior finite sample performance compared to other existing techniques.

Outlier Detection Using Gaussian Mixture Model Clustering to Optimize XGBoost for Credit Approval Prediction

Credit approval prediction is one of the critical challenges in the financial industry, where the accuracy and efficiency of credit decision-making can significantly affect business risk. This study proposes an outlier detection method using the Gaussian Mixture Model (GMM) combined with Extreme Gradient Boosting (XGBoost) to improve prediction accuracy. GMM is used to detect outliers with a probabilistic approach, allowing for finer-grained anomaly identification compared to distance- or density-based methods. Furthermore, the data cleaned through GMM is processed using XGBoost, a decision tree-based boosting algorithm that efficiently handles complex datasets. This study compares the performance of XGBoost with various outlier detection methods, such as LOF, CBLOF, DBSCAN, IF, and K-Means, as well as various other classification algorithms based on machine learning and deep learning. Experimental results show that the combination of GMM and XGBoost provides the best performance with an accuracy of 95.493%, a recall of 91.650%, and an AUC of 95.145%, outperforming other models in the context of credit approval prediction on an imbalanced dataset. The proposed method has been proven to reduce prediction errors and improve the model's reliability in detecting eligible credit applications.

GBMOD: A granular-ball mean-shift outlier detector

Outlier detection is a crucial data mining task involving identifying abnormal objects, errors, or emerging trends. Mean-shift-based outlier detection techniques evaluate the abnormality of an object by calculating the mean distance between the object and its k-nearest neighbors. However, in datasets with significant noise, the presence of noise in the k-nearest neighbors of some objects makes the model ineffective in detecting outliers. Additionally, the mean-shift outlier detection technique depends on finding the k-nearest neighbors of an object, which can be time-consuming. To address these issues, we propose a granular-ball computing-based mean-shift outlier detection method (GBMOD). Specifically, we first generate high-quality granular-balls to cover the data. By using the centers of granular-balls as anchors, the subsequent mean-shift process can effectively avoid the influence of noise points in the neighborhood. Then, outliers are detected based on the distance from the object to the displaced center of the granular-ball to which it belongs. Finally, the distance between the object and the shifted center of the granular-ball to which the object belongs is calculated, resulting in the outlier scores of objects. Subsequent experiments demonstrate the effectiveness, efficiency, and robustness of the method proposed in this paper.

Empirical Inferences Under Bayesian Framework to Identify Cellwise Outliers

Outliers are typically identified using frequentist methods. The data are classified as “outliers” or “not outliers” based on a test statistic that measures the magnitude of the difference between a value and the majority part of the data. The threshold for a data value to be an outlier is typically defined by the user. However, a subjective choice of the threshold increases the uncertainty associated with outlier status for each data value. A cellwise outlier detection algorithm named FuzzyHRT is used to automate the editing process in repeated surveys. This algorithm uses Bienaymé–Chebyshev’s inequality and fuzzy logic to detect four different types of outliers resulting from format inconsistencies, historical, tail, and relational anomalies. However, fuzzy logic is not suited for probabilistic reasoning behind the identification of anomalous cells. Bayesian methods are well suited for quantifying the uncertainty associated with the identification of outliers. Although, as suggested by the literature, there exist well-developed Bayesian methods for record-level outlier detection, Bayesian methods for identifying outliers within individual records (i.e., at the cell level) remain unexplored. This paper presents two approaches from the Bayesian perspective to study the uncertainty associated with identifying outliers. A Bayesian bootstrap approach is explored to study the uncertainty associated with the output scores from the FuzzyHRT algorithm. Empirical likelihoods in a Bayesian setting are also considered for probabilistic reasoning behind the identification of anomalous cells. NASS survey data for livestock and major crop yield (such as corn) are considered for comparing the performances of the two proposed approaches with recent cellwise outlier methods.

Integrated System Modal and Mistuning Identification for Integrally Bladed Rotors

Abstract The safety of Integrally Bladed Rotors (IBRs) is often assessed through bench-level vibration tests to measure amplification factors and back-out sector frequency deviations via mistuning identification (ID) algorithms. This process is usually composed of two separate steps. First, a system ID step is completed to identify system modal data. Then, this data is input into mistuning ID algorithms. Errors in identified modal data will then propagate to produce errors in the system's predicted mistuning. Obtaining robust mistuning estimates then requires larger quantities of accurate modal data. This effort seeks to attain accurate mistuning data by coupling the system and mistuning ID steps into a more parallel, versus serial, process that is capable of identifying many system modes. An iterative Polyreference-Least Squares Complex Frequency- domain (P-LSCF) algorithm finds modal data, and a mistuning ID algorithm obtains mistuning data at each iteration. An outlier detection method is proposed to remove spurious modes that cause erroneous mistuning results. Then, a weighted, least-squares regression approach is employed to remove the impact of sector-specific outlier data. This method reduces errors in identified mistuning parameters from the Fundamental Mistuning Model (FMM) ID algorithm. Furthermore, the approaches eliminate the need for users to determine true versus spurious system modes in each iteration of the P-LSCF algorithm, thus removing ambiguity. The developed approaches are tested on simulated and bench-top test data. Results show the efficacy of the developed approaches and their ability to account for uncertainty in mistuning parameters.

Provenance-based APT campaigns detection via masked graph representation learning

Advanced Persistent Threats (APTs) are well-planned, persistent, and highly stealthy cyberattacks designed to steal confidential information or disrupt specific target systems. Recent studies have used system audit logs to construct provenance graphs that describe system interactions to detect potentially malicious activities. Although they are effective, they still suffer from problems such as the need for a priori knowledge, lack of attack data, and high computational overhead that limit their application. In this paper, we propose a self-supervised learning-based APT detection model, APT-MGL, which learns the embedded representations of nodes through a graph mask self-encoder and transforms the detection problem into an outlier detection problem for malicious nodes. APT-MGL characterizes the behavior of nodes based on node type, action, and interaction frequency, and fuses the features through a multi-head self-attention mechanism. Then the node embedding is obtained by combining graph features and structural information using masked graph representation learning. Finally, the unsupervised outlier detection method is used to analyze the computed embeddings and obtain the final detection results. The experimental results show that APT-MGL outperforms existing monitoring models and achieves a small overhead.

Satterthwaite Approximation of the Distribution of SPE Scores: An R-Simulation-Based Improvement of the R-PCA-Based Outlier Detection Method

Satterthwaite Approximation of the Distribution of SPE Scores: An R-Simulation-Based Improvement of the R-PCA-Based Outlier Detection Method

Rockburst early-warning method based on time series prediction of multiple acoustic emission parameters

Rockburst early-warning is crucial for ensuring safety in deep underground engineering. Existing methods primarily focus on classifying rockburst grades, making it challenging to provide timely warnings. This paper proposes a novel rockburst early-warning framework based on time series prediction of acoustic emission (AE) parameters. Six AE parameters (rise time, count, duration, amplitude, absolute energy, and peak frequency) were identified as potential indicators for rockburst early-warning based on rockburst tests. A sliding window method was applied to process normalized AE data, calculating statistical parameters of the local duration. An LSTM-based time series prediction model was developed to forecast the future evolution of these AE parameters. This, in turn, enabled the establishment of a comprehensive multi-indicator early-warning system. The Isolation Forest (IF) algorithm, an outlier detection method, was used to determine the warning thresholds for each indicator. The CRITIC weighting method was employed to integrate the six rockburst indicators into a single early-warning coefficient (EC), with EC=100 signifying the warning trigger condition. The results demonstrate that the proposed framework effectively captures the evolution trends of AE parameters, enabling proactive early warnings. This approach addresses the limitations of existing methods, such as reliance on experience for threshold determination, lack of a clear basis for multi-indicator weights, and difficulty in quantifying early-warning trigger conditions. The framework provides a new perspective for rockburst early-warning systems.

Research on Outlier Detection Methods for Dam Monitoring Data Based on Post-Data Classification

Safety monitoring of hydraulic structures is a critical task in the field of hydraulic engineering construction. This study developed a method for preprocessing and classifying monitoring data for the identification of gross errors in hydraulic structures. By utilizing linear regression and wavelet analysis techniques, it effectively differentiated various waveform characteristics in data sets, such as Sinusoidal Wave Cyclical, Triangular Wave Cyclical, Seasonal Cyclical, and Weakly Cyclical growth types. In the experiments for gross error identification, the 3σ algorithm, K-medoids algorithm, and Isolation Forest algorithm were applied to test the data. The results showed that the K-medoids algorithm excelled in processing Sinusoidal Wave Cyclical Data Sets; the 3σ algorithm adapted better to Triangular Wave Cyclical Data Sets; the Isolation Forest algorithm performed well in handling data sets with significant anomalies or atypical fluctuations and excelled in scenarios with strong seasonality and large data fluctuations; and for complex Weakly Cyclical Growth Data Sets, all three algorithms were less effective, indicating the potential need for more advanced analysis methods or a combination of multiple techniques. Testing on actual engineering data further confirmed the importance of using specific gross error identification techniques for special data types after data set pre-classification, providing a more effective technical solution for the safety monitoring of hydraulic structures.

Adaptive threshold based outlier detection on IoT sensor data: A node-level perspective

The accuracy and reliability of IoT-based sensor networks depend on validating sensed data, including detecting outliers at the node level. This study proposes an online outlier detection approach using Multiple Linear Regression-based adaptive thresholds for real-time IoT/WSN sensor nodes. IoT sensors experience two outlier types: Errors, from sensor malfunctions or low battery, and Events, from sudden environmental changes. The Adaptive Threshold Based Outlier Detection (ATBOD) approach differentiates errors from events using an adaptive threshold that adjusts to real-time data patterns. Unlike existing methods that are used in literature, which lack automated model evolution and suffer from delays and high computational time, ATBOD enhances outlier detection sensitivity without increasing false alarms, which is crucial for efficient IoT sensor board operation. It also improves sensor board lifespan by discarding errors at the node level, preventing energy wastage from transmitting error data to the cloud. ATBOD outperforms existing algorithms, which are referenced for comparison, such as Enhanced Efficient Outlier Detection and Classification Approach (EEODCA), K Nearest Neighbor approximate outlier detection (KNN), and Modified Local Outlier Factor (LOF), in Error Detection Rate, Error False Positive Rate, and Energy Saving Ratio. These advancements represent a significant leap in performance, making ATBOD a superior method for real-time outlier detection in IoT sensor networks.