Outlier Detection Method Research Articles

Fast and explainable outlier detection in geotechnical database

ABSTRACT Outlier detection is an important task in data-driven geotechnics. It is noted that many existing methods for outlier detection (e.g. Bayesian learning, neural networks) may pose computational challenge for conventional geotechnical practitioners. Towards this aspect, this study proposes a simple, fast, and explainable method for detecting outliers in geotechnical database. The principle of the method is that for a data point to be labelled as a potential outlier, the probability of observing a data point as extreme as that value should be low. To account for outliers in left-tail and right-tail, the skewness of the dataset is incorporated, and an indicator referred to as outlier score ( > 0 ) is assigned to each data point. The method also provides another indicator (referred to as dimensional outlier score) to identify which dimensions/soil properties contribute to the outlierness; hence, it is explainable. The method doesn't require any time-consuming learning or sampling procedures; hence, it is quite fast and practitioner-friendly. Multiple numerical examples are utilised to demonstrate the capability first. Finally, four publicly available geotechnical databases are utilised to demonstrate the outlier detection task. The results suggest that the outliers identified using the proposed method can be meaningful from a geotechnical point of view.

Traffic Flow Outlier Detection for Smart Mobility Using Gaussian Process Regression Assisted Stochastic Differential Equations

Current methods for detecting outliers in traffic streaming data often struggle to capture real-time dynamic changes in traffic conditions and differentiate between genuine changes and anomalies. This study proposes a novel approach to outlier detection in traffic streaming data that effectively addresses stochasticity and uncertainty in observations. The proposed method utilizes Stochastic Differential Equations (SDEs) and Gaussian Process Regression (GPR). By employing SDEs, we can capture drift and diffusion estimates in traffic streaming data, providing a more comprehensive modeling of the data generation process. Integrating GPR allows precise Bayesian posterior inferences for outlier detection within the SDE framework. To improve practicality, we introduce a flexible threshold-setting mechanism using statistical testing to control the false positive rate. This adaptability helps strike a balance between model fitting and complexity in outlier detection. Compared to traditional SDE-based methods, our SDE-GPR outlier detection method demonstrates enhanced robustness and better adaptability to the complexities of traffic systems. This is evidenced through an empirical study using time series data collected in California, USA. Overall, this study introduces a more advanced and accurate approach to outlier detection in traffic streaming data, paving the way for improved real-time traffic condition monitoring and management.

Expanding the landscape of systemic sclerosis-related autoantibodies through RNA immunoprecipitation coupled with massive parallel sequencing

ObjectivesSystemic sclerosis (SSc)-related autoantibodies are widely used diagnostic and prognostic biomarkers. This study aimed to develop a new assay for detecting anti-ribonucleoprotein autoantibodies in SSc based on RNA immunoprecipitation (RNA IP) coupled with massive parallel sequencing. MethodsSerum samples and clinical data were collected from 307 SSc patients. Among these, 57 samples underwent analysis using a new protocol that combines RNA IP with massive parallel sequencing (RIP-Seq). Filtering strategies and statistical outlier detection methods were applied to select RNA molecules that could represent novel ribonucleoprotein autoantigens associated with SSc. ResultsAmong the 30,966 different RNA molecules identified by RIP-Seq in 57 SSc patients, 197 were ultimately selected. These included all RNA molecules previously identified by RNA IP, which were found to exhibit high counts almost exclusively in samples positive for the autoantibodies associated to the corresponding RNA molecule, indicating high sensitivity and specificity of the RIP-Seq technique. C/D box snoRNAs were the most abundant RNA type identified. The immunoprecipitation patterns of the detected C/D box snoRNAs varied among patients and could be associated with different clinical phenotypes. In addition, other ribonucleoproteins were identified, which could be potential targets for previously undescribed SSc-related autoantibodies. These include H/ACA box snoRNPs, vault complexes, mitochondrial tRNA synthetases, and 7SK snRNP. ConclusionA novel RIP-Seq assay has been developed to detect autoantibodies targeting ribonucleoprotein complexes in SSc patients. This method successfully identified RNA molecules associated with ribonucleoproteins known to be targeted by SSc-related autoantibodies, validating both the assay and the analysis strategy. Additionally, this approach uncovered RNA molecules associated with ribonucleoproteins that were not previously identified as targets of SSc patients’ sera, suggesting potential new autoantibody candidates in this disease.

Outlier Detection Using Gaussian Mixture Model Clustering to Optimize XGBoost for Credit Approval Prediction

Credit approval prediction is one of the critical challenges in the financial industry, where the accuracy and efficiency of credit decision-making can significantly affect business risk. This study proposes an outlier detection method using the Gaussian Mixture Model (GMM) combined with Extreme Gradient Boosting (XGBoost) to improve prediction accuracy. GMM is used to detect outliers with a probabilistic approach, allowing for finer-grained anomaly identification compared to distance- or density-based methods. Furthermore, the data cleaned through GMM is processed using XGBoost, a decision tree-based boosting algorithm that efficiently handles complex datasets. This study compares the performance of XGBoost with various outlier detection methods, such as LOF, CBLOF, DBSCAN, IF, and K-Means, as well as various other classification algorithms based on machine learning and deep learning. Experimental results show that the combination of GMM and XGBoost provides the best performance with an accuracy of 95.493%, a recall of 91.650%, and an AUC of 95.145%, outperforming other models in the context of credit approval prediction on an imbalanced dataset. The proposed method has been proven to reduce prediction errors and improve the model's reliability in detecting eligible credit applications.

Empirical Inferences Under Bayesian Framework to Identify Cellwise Outliers

Outliers are typically identified using frequentist methods. The data are classified as “outliers” or “not outliers” based on a test statistic that measures the magnitude of the difference between a value and the majority part of the data. The threshold for a data value to be an outlier is typically defined by the user. However, a subjective choice of the threshold increases the uncertainty associated with outlier status for each data value. A cellwise outlier detection algorithm named FuzzyHRT is used to automate the editing process in repeated surveys. This algorithm uses Bienaymé–Chebyshev’s inequality and fuzzy logic to detect four different types of outliers resulting from format inconsistencies, historical, tail, and relational anomalies. However, fuzzy logic is not suited for probabilistic reasoning behind the identification of anomalous cells. Bayesian methods are well suited for quantifying the uncertainty associated with the identification of outliers. Although, as suggested by the literature, there exist well-developed Bayesian methods for record-level outlier detection, Bayesian methods for identifying outliers within individual records (i.e., at the cell level) remain unexplored. This paper presents two approaches from the Bayesian perspective to study the uncertainty associated with identifying outliers. A Bayesian bootstrap approach is explored to study the uncertainty associated with the output scores from the FuzzyHRT algorithm. Empirical likelihoods in a Bayesian setting are also considered for probabilistic reasoning behind the identification of anomalous cells. NASS survey data for livestock and major crop yield (such as corn) are considered for comparing the performances of the two proposed approaches with recent cellwise outlier methods.

Integrated System Modal and Mistuning Identification for Integrally Bladed Rotors

Abstract The safety of Integrally Bladed Rotors (IBRs) is often assessed through bench-level vibration tests to measure amplification factors and back-out sector frequency deviations via mistuning identification (ID) algorithms. This process is usually composed of two separate steps. First, a system ID step is completed to identify system modal data. Then, this data is input into mistuning ID algorithms. Errors in identified modal data will then propagate to produce errors in the system's predicted mistuning. Obtaining robust mistuning estimates then requires larger quantities of accurate modal data. This effort seeks to attain accurate mistuning data by coupling the system and mistuning ID steps into a more parallel, versus serial, process that is capable of identifying many system modes. An iterative Polyreference-Least Squares Complex Frequency- domain (P-LSCF) algorithm finds modal data, and a mistuning ID algorithm obtains mistuning data at each iteration. An outlier detection method is proposed to remove spurious modes that cause erroneous mistuning results. Then, a weighted, least-squares regression approach is employed to remove the impact of sector-specific outlier data. This method reduces errors in identified mistuning parameters from the Fundamental Mistuning Model (FMM) ID algorithm. Furthermore, the approaches eliminate the need for users to determine true versus spurious system modes in each iteration of the P-LSCF algorithm, thus removing ambiguity. The developed approaches are tested on simulated and bench-top test data. Results show the efficacy of the developed approaches and their ability to account for uncertainty in mistuning parameters.

Satterthwaite Approximation of the Distribution of SPE Scores: An R-Simulation-Based Improvement of the R-PCA-Based Outlier Detection Method

Satterthwaite Approximation of the Distribution of SPE Scores: An R-Simulation-Based Improvement of the R-PCA-Based Outlier Detection Method

Provenance-based APT campaigns detection via masked graph representation learning

Advanced Persistent Threats (APTs) are well-planned, persistent, and highly stealthy cyberattacks designed to steal confidential information or disrupt specific target systems. Recent studies have used system audit logs to construct provenance graphs that describe system interactions to detect potentially malicious activities. Although they are effective, they still suffer from problems such as the need for a priori knowledge, lack of attack data, and high computational overhead that limit their application. In this paper, we propose a self-supervised learning-based APT detection model, APT-MGL, which learns the embedded representations of nodes through a graph mask self-encoder and transforms the detection problem into an outlier detection problem for malicious nodes. APT-MGL characterizes the behavior of nodes based on node type, action, and interaction frequency, and fuses the features through a multi-head self-attention mechanism. Then the node embedding is obtained by combining graph features and structural information using masked graph representation learning. Finally, the unsupervised outlier detection method is used to analyze the computed embeddings and obtain the final detection results. The experimental results show that APT-MGL outperforms existing monitoring models and achieves a small overhead.

Rockburst early-warning method based on time series prediction of multiple acoustic emission parameters

Rockburst early-warning is crucial for ensuring safety in deep underground engineering. Existing methods primarily focus on classifying rockburst grades, making it challenging to provide timely warnings. This paper proposes a novel rockburst early-warning framework based on time series prediction of acoustic emission (AE) parameters. Six AE parameters (rise time, count, duration, amplitude, absolute energy, and peak frequency) were identified as potential indicators for rockburst early-warning based on rockburst tests. A sliding window method was applied to process normalized AE data, calculating statistical parameters of the local duration. An LSTM-based time series prediction model was developed to forecast the future evolution of these AE parameters. This, in turn, enabled the establishment of a comprehensive multi-indicator early-warning system. The Isolation Forest (IF) algorithm, an outlier detection method, was used to determine the warning thresholds for each indicator. The CRITIC weighting method was employed to integrate the six rockburst indicators into a single early-warning coefficient (EC), with EC=100 signifying the warning trigger condition. The results demonstrate that the proposed framework effectively captures the evolution trends of AE parameters, enabling proactive early warnings. This approach addresses the limitations of existing methods, such as reliance on experience for threshold determination, lack of a clear basis for multi-indicator weights, and difficulty in quantifying early-warning trigger conditions. The framework provides a new perspective for rockburst early-warning systems.

Research on Outlier Detection Methods for Dam Monitoring Data Based on Post-Data Classification

Safety monitoring of hydraulic structures is a critical task in the field of hydraulic engineering construction. This study developed a method for preprocessing and classifying monitoring data for the identification of gross errors in hydraulic structures. By utilizing linear regression and wavelet analysis techniques, it effectively differentiated various waveform characteristics in data sets, such as Sinusoidal Wave Cyclical, Triangular Wave Cyclical, Seasonal Cyclical, and Weakly Cyclical growth types. In the experiments for gross error identification, the 3σ algorithm, K-medoids algorithm, and Isolation Forest algorithm were applied to test the data. The results showed that the K-medoids algorithm excelled in processing Sinusoidal Wave Cyclical Data Sets; the 3σ algorithm adapted better to Triangular Wave Cyclical Data Sets; the Isolation Forest algorithm performed well in handling data sets with significant anomalies or atypical fluctuations and excelled in scenarios with strong seasonality and large data fluctuations; and for complex Weakly Cyclical Growth Data Sets, all three algorithms were less effective, indicating the potential need for more advanced analysis methods or a combination of multiple techniques. Testing on actual engineering data further confirmed the importance of using specific gross error identification techniques for special data types after data set pre-classification, providing a more effective technical solution for the safety monitoring of hydraulic structures.

Adaptive threshold based outlier detection on IoT sensor data: A node-level perspective

The accuracy and reliability of IoT-based sensor networks depend on validating sensed data, including detecting outliers at the node level. This study proposes an online outlier detection approach using Multiple Linear Regression-based adaptive thresholds for real-time IoT/WSN sensor nodes. IoT sensors experience two outlier types: Errors, from sensor malfunctions or low battery, and Events, from sudden environmental changes. The Adaptive Threshold Based Outlier Detection (ATBOD) approach differentiates errors from events using an adaptive threshold that adjusts to real-time data patterns. Unlike existing methods that are used in literature, which lack automated model evolution and suffer from delays and high computational time, ATBOD enhances outlier detection sensitivity without increasing false alarms, which is crucial for efficient IoT sensor board operation. It also improves sensor board lifespan by discarding errors at the node level, preventing energy wastage from transmitting error data to the cloud. ATBOD outperforms existing algorithms, which are referenced for comparison, such as Enhanced Efficient Outlier Detection and Classification Approach (EEODCA), K Nearest Neighbor approximate outlier detection (KNN), and Modified Local Outlier Factor (LOF), in Error Detection Rate, Error False Positive Rate, and Energy Saving Ratio. These advancements represent a significant leap in performance, making ATBOD a superior method for real-time outlier detection in IoT sensor networks.

Outlier detection using local density and global structure

Outlier detection, a critical process for enhancing data quality, has been applied to a broad spectrum of real-world applications. This study introduces a novel and effective outlier detection method that incorporates both local density and global structural information. Specifically, we employ the concept of potential energy, a well-established principle in physics, to represent local densities or neighbor distributions of data. Additionally, we also exploit the notion of hubness, a measure explored in network science, to further capture global structural information of data. To facilitate this, we construct a strongly connected digraph by introducing a virtual node along with an associated set of virtual edges. This digraph underpins a tailored Markov random walk process that is used to estimate the hubness scores of data objects. Subsequently, the importance degrees of data objects are estimated according to their potential energies and hubness scores. The data objects with lower importance degrees are deemed to be outliers. Extensive experiments conducted on thirteen publicly accessible real-world datasets against nine popular outlier detection algorithms show that the proposed method achieved encouraging and competitive performance in comparing the state-of-the-art outlier detection algorithms.

Counterfeit banknote identification based on outliers detection methods

Counterfeit banknotes in circulation in the economy are a long-standing and recurring problem. Technological improvements are applied to produce increasingly convincing counterfeit banknotes. In contrast, sophisticated strategies to identify counterfeit notes remain an up-to-date and fascinating research topic of great interest. The literature presents several techniques for detecting counterfeit banknotes. In this study, outlier detection methodologies are presented as adapted mechanisms to assist in the fight against counterfeit banknotes. This study addresses and tests multivariate outlier detection techniques' efficiency in identifying counterfeit banknotes. A comprehensive computational experiment proves the great adaptability of the methods to multivariate outliers in the identification of counterfeit banknotes. In particular, the Data-driven Cluster Analysis Method (DDCAM) technique presented promising and optimistic results through several quality and performance measurement metrics, suggesting a hopeful future for detecting counterfeit banknotes.

A Novel Data-Driven Approach with a Long Short-Term Memory Autoencoder Model with a Multihead Self-Attention Deep Learning Model for Wind Turbine Converter Fault Detection

The imminent depletion of oil resources and increasing environmental pollution have driven the use of clean energy, particularly wind energy. However, wind turbines (WTs) face significant challenges, such as critical component failures, which can cause unexpected shutdowns and affect energy production. To address this challenge, we analyzed the Supervisory Control and Data Acquisition (SCADA) data to identify significant differences between the relationship of variables based on data reconstruction errors between actual and predicted values. This study proposes a hybrid short- and long-term memory autoencoder model with multihead self-attention (LSTM-MA-AE) for WT converter fault detection. The proposed model identifies anomalies in the data by comparing the reconstruction errors of the variables involved. However, more is needed. To address this model limitation, we developed a fault prediction system that employs an adaptive threshold with an Exponentially Weighted Moving Average (EWMA) and a fixed threshold. This system analyzes the anomalies of several variables and generates fault warnings in advance time. Thus, we propose an outlier detection method through data preprocessing and unsupervised learning, using SCADA data collected from a wind farm located in complex terrain, including real faults in the converter. The LSTM-MA-AE is shown to be able to predict the converter failure 3.3 months in advance, and with an F1 greater than 90% in the tests performed. The results provide evidence of the potential of the proposed model to improve converter fault diagnosis with SCADA data in complex environments, highlighting its ability to increase the reliability and efficiency of WTs.

Outlier Detection in Streaming Data for Telecommunications and Industrial Applications: A Survey

Streaming data are present all around us. From traditional radio systems streaming audio to today’s connected end-user devices constantly sending information or accessing services, data are flowing constantly between nodes across various networks. The demand for appropriate outlier detection (OD) methods in the fields of fault detection, special events detection, and malicious activities detection and prevention is not only persistent over time but increasing, especially with the recent developments in Telecommunication systems such as Fifth Generation (5G) networks facilitating the expansion of the Internet of Things (IoT). The process of selecting a computationally efficient OD method, adapted for a specific field and accounting for the existence of empirical data, or lack thereof, is non-trivial. This paper presents a thorough survey of OD methods, categorized by the applications they are implemented in, the basic assumptions that they use according to the characteristics of the streaming data, and a summary of the emerging challenges, such as the evolving structure and nature of the data and their dimensionality and temporality. A categorization of commonly used datasets in the context of streaming data is produced to aid data source identification for researchers in this field. Based on this, guidelines for OD method selection are defined, which consider flexibility and sample size requirements and facilitate the design of such algorithms in Telecommunications and other industries.

Comparison of energy consumption prediction models for air conditioning at different time scales for large public buildings

Large public buildings have complex functions and high air conditioning energy consumption, and the prediction of energy consumption for air conditioning in different time scales can realise different energy saving management needs. Taking a large public building in Guangzhou as a research case, the long and short-term memory neural network (LSTM) is used as the basis to establish the machine learning model for energy consumption prediction for air conditioning at different time scales in terms of 10 min, hourly and daily, and a comparative analysis is carried out. The original data were analysed and processed by Z-Score and Local Outlier Factor (LOF) outlier detection methods. The relationship between meteorological parameters for Typical Meteorological Year (TMY) and the actual local meteorological parameters and energy consumption for air conditioning was analysed using linear regression, Pearson and Spearman correlation coefficients, taking meteorological parameters as input variables. The results show that comparing energy consumption for air conditioning with the meteorological parameters in TMY and actual local meteorological parameters, the actual meteorological parameters had a stronger correlation energy consumption for air conditioning. When the actual local meteorological parameters are taken as input variables, the mean absolute percentage error (MAPE) is reduced by about 0.85 %, and the coefficient of determination (R2) is increased by about 0.01. At the same time, the MAPE and R2 of the 10-min energy consumption prediction model are about 6.43 % and 0.9738, respectively, the MAPE and R2 of the hourly prediction model are about 9.29 % and 0.9568, respectively, while the MAPE and R2 of the daily prediction model are about 25.76 % and 0.7998. Meanwhile, an improved energy consumption prediction model is proposed to reduce the daily energy consumption prediction error to 6.36 % for MAPE, and the R2 of this model is 0.9927.

Application of the outlier detection method for web-based blood glucose level monitoring system

Recent advancements in biosensors have empowered individuals with diabetes to autonomously monitor their blood glucose levels through continuous glucose monitoring (CGM) sensors. Nevertheless, the data collected from these sensors may occasionally include outliers due to the inherent imperfections of the sensor devices. Consequently, the identification of these outliers is critical to determine whether blood glucose levels deviate significantly from the norm, necessitating further action. This study employs an outlier detection approach based on the 3-sigma method and the interquartile range (IQR), along with the application of the Winsorizing technique to correct the identified outliers. Additionally, a web-based system for visualizing blood glucose levels is developed, utilizing both outlier detection methods. In order to assess the system's performance, two types of testing are conducted: black box testing and load testing. The results of black box testing indicate that all test scenarios operate as anticipated. As for the load testing response times, it is observed that the 3-sigma visualization page loads an average of 606.75 milliseconds faster compared to the IQR visualization page. This study's outcomes are expected to enhance data quality, enhance the precision of analyses, and facilitate more informed decision-making by identifying and addressing extreme data points.

Outlier detection and selection of representative fluid samples using machine learning: a case study of Iranian oil fields

During the development of a field, many fluid samples are taken from wells. Selecting a robust fluid sample as the reservoir representative helps to have a better field characterization, reliable reservoir simulation, valid production forecast, efficient well placement and finally achieving optimized ultimate recovery. First, this paper aims to detect and separate the samples that have been collected under poor conditions or analyzed in a non-standard way. Moreover, it introduces a novel ranking method to score the samples based on the amount of coordination with other fluid samples in the region. The dataset includes 136 fluid samples from five reservoirs in Iranian fields, each of them consisting of 21 key parameters. Five acknowledged machine learning based anomaly detection techniques are implemented to compare fluid samples and detect those whose results deviate from others, indicating non-standard samples. To ensure the proper detection of outlier data, the results are compared with the traditional validation method of gas-oil ratio estimation. All five outlier detection methods demonstrate acceptable performance with average accuracy of 79% compared to traditional validation. Furthermore, the fluid samples with the highest scores in scoring-based algorithms are introduced as the best reservoir’s representative fluid. Finally, fuzzy logic is used to obtain a final score for each sample, taking the results of the six methods as input and ranking the samples based on their output score. The study confirms the robustness of the novel approach for fluid validation using outlier detection techniques and the value of machine learning and fuzzy logic for sample ranking, excelling in considering all critical fluid parameters simultaneously over traditional methods.

Enhancing Community E-Commerce Repurchase Prediction Through Information Entropy Analysis

In this study, we delve into the prediction and recommendation algorithm for user repurchase behavior within the community e-commerce landscape, leveraging the framework of information entropy. By analyzing consumer behavior data from a product-centered perspective, we observe that the majority of consumers engage in approximately 0-8 information search clicks before making a purchase, primarily focusing on product details, with an average of about 4 browsing interactions. Subsequently, a minimal portion of consumers exhibit 0-5 instances of preference behavior after browsing activities. Similarly, consumers who add items to their shopping carts typically engage in this behavior from 0 to 4 times, showcasing limited participation. Leveraging attribute probability, information entropy serves as a driving force for predicting community e-commerce users' repurchase behavior, contrasting with traditional outlier detection methods and underscoring the advantages of information entropy.

Unsupervised outlier detection using random subspace and subsampling ensembles of Dirichlet process mixtures

Probabilistic mixture models are recognized as effective tools for unsupervised outlier detection owing to their interpretability and global characteristics. Among these, Dirichlet process mixture models stand out as a strong alternative to conventional finite mixture models for both clustering and outlier detection tasks. Unlike finite mixture models, Dirichlet process mixtures are infinite mixture models that automatically determine the number of mixture components based on the data. Despite their advantages, the adoption of Dirichlet process mixture models for unsupervised outlier detection has been limited by challenges related to computational inefficiency and sensitivity to outliers in the construction of outlier detectors. Additionally, Dirichlet process Gaussian mixtures struggle to effectively model non-Gaussian data with discrete or binary features. To address these challenges, we propose a novel outlier detection method that utilizes ensembles of Dirichlet process Gaussian mixtures. This unsupervised algorithm employs random subspace and subsampling ensembles to ensure efficient computation and improve the robustness of the outlier detector. The ensemble approach further improves the suitability of the proposed method for detecting outliers in non-Gaussian data. Furthermore, our method uses variational inference for Dirichlet process mixtures, which ensures both efficient and rapid computation. Empirical analyses using benchmark datasets demonstrate that our method outperforms existing approaches in unsupervised outlier detection.