Outlier Detection Approach Research Articles

An Outlier Detection Approach to Recognize the Sources of a Process Failure within a Multivariate Poisson Process

An Outlier Detection Approach to Recognize the Sources of a Process Failure within a Multivariate Poisson Process

Adaptive threshold based outlier detection on IoT sensor data: A node-level perspective

The accuracy and reliability of IoT-based sensor networks depend on validating sensed data, including detecting outliers at the node level. This study proposes an online outlier detection approach using Multiple Linear Regression-based adaptive thresholds for real-time IoT/WSN sensor nodes. IoT sensors experience two outlier types: Errors, from sensor malfunctions or low battery, and Events, from sudden environmental changes. The Adaptive Threshold Based Outlier Detection (ATBOD) approach differentiates errors from events using an adaptive threshold that adjusts to real-time data patterns. Unlike existing methods that are used in literature, which lack automated model evolution and suffer from delays and high computational time, ATBOD enhances outlier detection sensitivity without increasing false alarms, which is crucial for efficient IoT sensor board operation. It also improves sensor board lifespan by discarding errors at the node level, preventing energy wastage from transmitting error data to the cloud. ATBOD outperforms existing algorithms, which are referenced for comparison, such as Enhanced Efficient Outlier Detection and Classification Approach (EEODCA), K Nearest Neighbor approximate outlier detection (KNN), and Modified Local Outlier Factor (LOF), in Error Detection Rate, Error False Positive Rate, and Energy Saving Ratio. These advancements represent a significant leap in performance, making ATBOD a superior method for real-time outlier detection in IoT sensor networks.

Application of the outlier detection method for web-based blood glucose level monitoring system

Recent advancements in biosensors have empowered individuals with diabetes to autonomously monitor their blood glucose levels through continuous glucose monitoring (CGM) sensors. Nevertheless, the data collected from these sensors may occasionally include outliers due to the inherent imperfections of the sensor devices. Consequently, the identification of these outliers is critical to determine whether blood glucose levels deviate significantly from the norm, necessitating further action. This study employs an outlier detection approach based on the 3-sigma method and the interquartile range (IQR), along with the application of the Winsorizing technique to correct the identified outliers. Additionally, a web-based system for visualizing blood glucose levels is developed, utilizing both outlier detection methods. In order to assess the system's performance, two types of testing are conducted: black box testing and load testing. The results of black box testing indicate that all test scenarios operate as anticipated. As for the load testing response times, it is observed that the 3-sigma visualization page loads an average of 606.75 milliseconds faster compared to the IQR visualization page. This study's outcomes are expected to enhance data quality, enhance the precision of analyses, and facilitate more informed decision-making by identifying and addressing extreme data points.

Comparison of outlier detection approaches for compressive strength of cement-based mortars

EN 196-1 states that the compressive strength result of testing of standard mortar specimens corresponds to the average of the six individual measurements. If one of the six individual measurements varies more than ±10 % from the average, it must be excluded, and a new average is calculated with the five remaining values. This methodology might be interesting for quality control or characterization of cement; however, it may not be the best approach for research or scientific purposes. Applying EN 196-1 to non-standard cement-based mortars with multiple and even unconventional supplementary cementitious materials may induce even more variation in the result. Also, mixtures with higher mechanical strength generally present the highest absolute variation. The main goal of the present work is to compare different methodologies, such as Tails of Normal Distribution, Thompson-Tau, Box-plot, T-test, Dixon Criteria, and other methods provided by engineering standards, to evaluate the existence of discrepant values and the effect of their exclusion (or not) on compressive strength experimental values, namely for science purposes. A sample of compressive strength of 338 mortar specimens was used The main findings revealed that excluding data points as a result of applying 8 of the 13 methods under study did not affect the mechanical strength result by more than ±5 %. The method described in the Brazilian standard that guides preparation, control, receipt, and acceptance for Portland cement concrete, would invalidate almost 46 % of all mixtures produced and did not prove to be reliable. Simplified methods of excluding outliers in relative or absolute percentages of 6 % around the average value can be considered an interesting option for professionals involved in laboratory testing of building materials. Otherwise, the authors would recommend applying the either Dixon Criteria or T test of the ASTM E178 for scientific purposes.

Implementing robust outlier detection to enhance estimation accuracy of GNSS-IR based seasonal snow depth retrievals

ABSTRACT Monitoring snow depth variations aligned with the seasonal cycle is crucial for studies on climate change and its impacts due to global warming. Especially in terms of snow hydrology, the determination and continuous monitoring of snow depth to determine snow water equivalent is a priority in climate, water science, drought, flood, and inundation prediction studies, particularly in the water cycle. Recently, the so-called Global Navigation Satellite Systems – Interferometric Reflectometry (GNSS-IR) method enables to extract environmental radiometric and geometric characteristics of surface where the signals transmitted from the satellites reflect. The reflected signal follows an extra path than the direct one and causes a major error source for accurate point positioning but a useful tool for sensing the environment. However, outliers remain a challenge in long-term GNSS-IR estimations. In this study, we proposed a median-based robust outlier detection (ROD) approach for identifying outliers in long-term GNSS-IR snow depth estimations. To validate our approach, we analysed 5-year GNSS L1 SNR and L2 SNR data from 1 January 2015 to 31 December 2019 provided by AB33 and AB39 GNSS stations in Alaska, U.S.A. using the GNSS-IR method. We validated GNSS-IR estimations using snow depth measurements from the Coldfoot and Fort Yukon stations in the SNOTEL network. Applying ROD to long-term snow depth estimates increased the highest correlation from 91.58% to 94.93%, and reduced the lowest RMSE from 8.6 cm to 6.7 cm. In addition, the improvement rates calculated to assess the contribution of ROD to the results showed improvements of up to 6.9% in correlation and 26.1% in RMSE. Overall, the results demonstrate that ROD can be effectively used to detect outliers in long-term GNSS-IR snow depth time series.

Consistent Robust Analytical Approach for Outlier Detection in Multivariate Data using Isolation Forest and Local Outlier Factor

Outlier detection in real-time from multivariate streaming data is an important research subject in numerous areas. The new presentation of gradual Neighborhood Anomaly Variable (iLOF) and its variations has acquired consideration for their high recognition execution in information streams with evolving circulations. This paper presents a new intelligent exception location framework called include rich intelligent anomaly discovery, which integrates human interaction into the detection process to improve performance and simplify detection. It offers interactive mechanism that allows for instinctive client cooperation during every vital stage of the fundamental anomaly recognition calculation, for example, thick cell choice, area mindful distance thresholding, and last top exception approval. This approach helps resolve the challenge of specifying key parameters like density and distance thresholds in other outlier detection methods. Additionally, the system proposes an innovative optimization method to enhance grid-based space partitioning. The Local Outlier Factor (LOF) measures the level of outlierness of each occasion in light of the conveyance thickness in the dataset. Higher LOF values indicate a higher likelihood of an instance being an outlier. Instances with LOF values above a set threshold are identified as outliers. The calculation of LOF values involves several steps, which are detailed in original articles for further reference.

VOD: Vision-Based Building Energy Data Outlier Detection

Outlier detection plays a critical role in building operation optimization and data quality maintenance. However, existing methods often struggle with the complexity and variability of building energy data, leading to poorly generalized and explainable results. To address the gap, this study introduces a novel Vision-based Outlier Detection (VOD) approach, leveraging computer vision models to spot outliers in the building energy records. The models are trained to identify outliers by analyzing the load shapes in 2D time series plots derived from the energy data. The VOD approach is tested on four years of workday time-series electricity consumption data from 290 commercial buildings in the United States. Two distinct models are developed for different usage purposes, namely a classification model for broad-level outlier detection and an object detection model for the demands of precise pinpointing of outliers. The classification model is also interpreted via Grad-CAM to enhance its usage reliability. The classification model achieves an F1 score of 0.88, and the object detection model achieves an Average Precision (AP) of 0.84. VOD is a very efficient path to identifying energy consumption outliers in building operations, paving the way for the enhancement of building energy data quality, operation efficiency, and energy savings.

Kernel Function Selection for FISVDD Applied to Outliers Detection Application for Energy Time Series Datasets

Forecasting energy consumption is crucial for maintaining stability in the energy supply-demand balance. This process requires accurate data, necessitating deep preprocessing steps, with outlier detection being a pivotal task. While most existing methods are tailored for batch learning, smart grid data collection operates continuously, demanding online algorithms for effective data cleaning. This paper proposes an analytical study emphasizing the significance of selecting an appropriate kernel function for data description methods and its impact on forecasting efficiency. The Fast Incremental Support Vector Data Description (FISVDD) algorithm is chosen as a kernel-based approach for outlier detection in time series datasets. The evaluation involves different kernel functions on two datasets: Electrical Grid Stability Simulated Data and Individual Household Electric Power Consumption Dataset. Criteria such as the objective function and data distribution are considered to determine the most suitable kernel function. FISVDD's performance with the right kernel function is compared with other outlier detection methods, and the results are fed into a Gated Recurrent Unit model to visualize the impact on forecasting. Mean Squared Error evaluation reveals that FISVDD with the appropriate kernel function yields the best results, underscoring the strength of data description methods with the appropriate kernel function.

Outlier detection in spatial error models using modified thresholding-based iterative procedure for outlier detection approach

BackgroundOutliers, data points that significantly deviate from the norm, can have a substantial impact on statistical inference and provide valuable insights in data analysis. Multiple methods have been developed for outlier detection, however, almost all available approaches fail to consider the spatial dependence and heterogeneity in spatial data. Spatial data has diverse formats and semantics, requiring specialized outlier detection methodology to handle these unique properties. For now, there is limited research exists on robust spatial outlier detection methods designed specifically under the spatial error model (SEM) structure.MethodWe propose the Spatial-Θ-Iterative Procedure for Outlier Detection (Spatial-Θ-IPOD), which utilizes a mean-shift vector to identify outliers within the SEM. Our method enables an effective detection of spatial outliers while also providing robust coefficient estimates. To assess the performance of our approach, we conducted extensive simulations and applied it to a real-world empirical study using life expectancy data from multiple countries.ResultsSimulation results showed that the masking and JD (Joint Detection) indicators of our Spatial-Θ-IPOD method outperformed several commonly used methods, even in high-dimensional scenarios, demonstrating stable performance. Conversely, the Θ-IPOD method proved to be ineffective in detecting outliers when spatial correlation was present. Moreover, our model successfully provided reliable coefficient estimation alongside outlier detection. The proposed method consistently outperformed other models (both robust and non-robust) in most cases. In the empirical study, our proposed model successfully detected outliers and provided valuable insights in the modeling process.ConclusionsOur proposed Spatial-Θ-IPOD offers an effective solution for detecting spatial outliers for SEM while providing robust coefficient estimates. Notably, our approach showcases its relative superiority even in the presence of high leverage points. By successfully identifying outliers, our method enhances the overall understanding of the data and provides valuable insights for further analysis.

SEAOP: a statistical ensemble approach for outlier detection in quantitative proteomics data.

Quality control in quantitative proteomics is a persistent challenge, particularly in identifying and managing outliers. Unsupervised learning models, which rely on data structure rather than predefined labels, offer potential solutions. However, without clear labels, their effectiveness might be compromised. Single models are susceptible to the randomness of parameters and initialization, which can result in a high rate of false positives. Ensemble models, on the other hand, have shown capabilities in effectively mitigating the impacts of such randomness and assisting in accurately detecting true outliers. Therefore, we introduced SEAOP, a Python toolbox that utilizes an ensemble mechanism by integrating multi-round data management and a statistics-based decision pipeline with multiple models. Specifically, SEAOP uses multi-round resampling to create diverse sub-data spaces and employs outlier detection methods to identify candidate outliers in each space. Candidates are then aggregated as confirmed outliers via a chi-square test, adhering to a 95% confidence level, to ensure the precision of the unsupervised approaches. Additionally, SEAOP introduces a visualization strategy, specifically designed to intuitively and effectively display the distribution of both outlier and non-outlier samples. Optimal hyperparameter models of SEAOP for outlier detection were identified by using a gradient-simulated standard dataset and Mann-Kendall trend test. The performance of the SEAOP toolbox was evaluated using three experimental datasets, confirming its reliability and accuracy in handling quantitative proteomics.

Boosting edgeR (Robust) by dealing with missing observations and gene-specific outliers in RNA-Seq profiles and its application to explore biomarker genes for diagnosis and therapies of ovarian cancer

The edgeR (Robust) is a popular approach for identifying differentially expressed genes (DEGs) from RNA-Seq profiles. However, it shows weak performance against gene-specific outliers and is unable to handle missing observations. To address these issues, we proposed a pre-processing approach of RNA-Seq count data by combining the iLOO-based outlier detection and random forest-based missing imputation approach for boosting the performance of edgeR (Robust). Both simulation and real RNA-Seq count data analysis results showed that the proposed edgeR (Robust) outperformed than the conventional edgeR (Robust). To investigate the effectiveness of identified DEGs for diagnosis, and therapies of ovarian cancer (OC), we selected top-ranked 12 DEGs (IL6, XCL1, CXCL8, C1QC, C1QB, SNAI2, TYROBP, COL1A2, SNAP25, NTS, CXCL2, and AGT) and suggested hub-DEGs guided top-ranked 10 candidate drug-molecules for the treatment against OC. Hence, our proposed procedure might be an effective computational tool for exploring potential DEGs from RNA-Seq profiles for diagnosis and therapies of any disease.

Wind turbine condition monitoring based on three fitted performance curves

AbstractBased on SCADA data, this study aims at fitting three performance curves (PCs), power curve, pitch angle curve, and rotor speed curve, to accurately describe the normal behaviour of a wind turbine (WT) for performance monitoring and identification of anomalous signals. The fitting accuracy can be undesirably affected by erroneous SCADA data. Hence, outliers generated from raw SCADA data should be removed to mitigate the prediction inaccuracy, so various outlier detection (OD) approaches are compared in terms of area under the curve (AUC) and mean average precision (mAP). Among them, a novel unsupervised SVM‐KNN model, integrated by support vector machine (SVM) and k nearest neighbour (KNN), is the optimum detector for PC refinements. Based on the refined data by the SVM‐KNN detector, several common nonparametric regressors have largely improved their prediction accuracies on pitch angle and rotor speed curves from roughly 86% and 90.6%, respectively, (raw data) to both 99% (refined data). Noticeably, under the SVM‐KNN refinement, the errors have been reduced by roughly five times and 10 times for pitch angle and rotor speed predictions, respectively. Ultimately, bootstrapped prediction interval is applied to conduct the uncertainty analysis of the optimal predictive regression model, reinforcing the performance monitoring and anomaly detection.

Quantile-Based Maximum Likelihood Training for Outlier Detection

Discriminative learning effectively predicts true object class for image classification. However, it often results in false positives for outliers, posing critical concerns in applications like autonomous driving and video surveillance systems. Previous attempts to address this challenge involved training image classifiers through contrastive learning using actual outlier data or synthesizing outliers for self-supervised learning. Furthermore, unsupervised generative modeling of inliers in pixel space has shown limited success for outlier detection. In this work, we introduce a quantile-based maximum likelihood objective for learning the inlier distribution to improve the outlier separation during inference. Our approach fits a normalizing flow to pre-trained discriminative features and detects the outliers according to the evaluated log-likelihood. The experimental evaluation demonstrates the effectiveness of our method as it surpasses the performance of the state-of-the-art unsupervised methods for outlier detection. The results are also competitive compared with a recent self-supervised approach for outlier detection. Our work allows to reduce dependency on well-sampled negative training data, which is especially important for domains like medical diagnostics or remote sensing.

Random clustering-based outlier detector

Outlier detection is one of the most important issues in contemporary data analysis. At present, many methods are employed for anomaly and outlier detection, but there is still no universal tool that delivers a high degree of efficiency. In this study, we present a novel approach for outlier detection based on the skillful use of the law of large numbers. The main idea of the proposed solution consists of the random clustering of the elements of the analyzed set. Then, those elements that are sufficiently distant from the random cluster centers are marked as outliers. The proposed approach, besides being highly effective, is also very intuitive. The results of the conducted numerical experiments confirm the high degree of effectiveness of the proposed method, with the measures of accuracy and precision reaching a value of 1. The indisputable advantages of this novel approach for outlier detection are the simplicity of interpretation and the possibility of its modification by people who may lack an extensive experience in data analysis. The effectiveness of the proposed method was compared with other recognized techniques in detecting outliers within both artificially generated and empirical data sets.

SA-O2DCA: Seasonal Adapted Online Outlier Detection and Classification Approach for WSN

SA-O2DCA: Seasonal Adapted Online Outlier Detection and Classification Approach for WSN

A rigorous real-time acoustic positioning method for ocean bottom seismic exploration

The conventional technique for positioning seafloor geophones in ocean bottom seismic exploration encounters several challenges, including the significant impact of outliers on positioning results, underutilization of high-precision observations, and low efficiency in real-time data processing. These issues inevitably affect the quality of seismic exploration outcomes. To address these challenges and enhance the accuracy of geophone positioning, this paper proposes a rigorous real-time acoustic positioning method for geophones based on sequential adjustment and Baarda's outlier detection approach. The proposed method comprises three key steps: grouping the original acoustic observations, constructing the intra-group acoustic positioning model, and synthesizing the positioning results across the different groups. The validity and practicality of this approach are confirmed through a simulation experiment as well as the field experiment conducted in the Bohai Sea, China. The results demonstrate that the proposed method effectively eliminates outliers in the original observations and maximizes the utilization of high-quality observations. Compared to traditional acoustic positioning methods, it significantly reduces positioning errors from meters to decimeters, and in some cases can achieve centimeter-level precision. When the sound velocity profile in the operating sea area is measured, the method can attain the posterior standard deviation at the millimeter level and positioning errors within 10 cm. When the sound velocity profile is unknown, the method can achieve the posterior standard deviation at centimeter-level and positioning errors of approximately 20 cm.

TEDA-RLS: A TinyML Incremental Learning Approach for Outlier Detection and Correction

TEDA-RLS: A TinyML Incremental Learning Approach for Outlier Detection and Correction

Hyperplane tree-based data mining with a multi-functional memristive crossbar array.

This study explores the stochastic and binary switching behaviors of a Ta/HfO2/RuO2 memristor to implement a combined data mining approach for outlier detection and data clustering algorithms in a multi-functional memristive crossbar array. The memristor switches stochastically with high state dispersion in the stochastic mode and deterministically between two states with low dispersion in the binary mode, while they can be controlled by varying operating voltages. The stochastic mode facilitates the parallel generation of random hyperplanes in a tree structure, used to compress spatial information of the dataset in the Euclidian space into binary format, still retaining sufficient spatial features. The ensemble effect from multiple trees improved the classification performance. The binary mode facilitates parallel Hamming distance calculation of the binary codes containing spatial information, which measures similarity. These two modes enable efficient implementation of the newly proposed minority-based outlier detection method and modified K-means method on the same hardware. Array measurements and hardware simulations investigate various hyperparameters' impact and validate the proposed methods with practical datasets. The proposed methods show linear O(n) time complexity and high energy efficiency, consuming <1% of the energy compared to digital computing with conventional algorithms while demonstrating software-comparable performance in both tasks.

Comparison of outlier detection approaches in a Smart Cities sensor data context

Abstract This study examines outlier detection in time-series sensor data from PurpleAir low-cost sensors in Athens, Greece. Focusing on key environmental parameters such as temperature, humidity, and particulate matter (PM) levels, the study utilizes the Interquartile Range (IQR) and Generalized Extreme Studentized Deviate (GESD) methods on hourly and daily basis. GESD detected more outliers than IQR, most of them in PM, while temperature and humidity data had fewer outliers; applying filters before outlier detection and adjusting alpha values based on time scales were crucial, and outliers significantly affected spatial interpolation, emphasizing the need for spatial statistics in smart city air quality management.

A new particle-swarm-optimization-assisted deep transfer learning framework with applications to outlier detection in additive manufacturing

In wire arc additive manufacturing (WAAM), the electric arc is an essential part of the welding equipment, which serves as the heat source and is directed by the current and voltage. The working status of the electric arc is an important factor in determining the quality of the fabricated components. During the welding process, the current and voltage may change abruptly due to some abnormalities in the operating conditions, which may affect the working status and thereby affect the quality of products. Such abnormal changes in the current and voltage can be treated as outliers. In order to identify outliers in current and voltage to further improve the welding process, in this paper, a novel deep-transfer-learning-embedded outlier detection approach is developed for WAAM. A new domain adaptation strategy is designed where the cross-domain discrepancies of the marginal distribution and conditional distribution are minimized. Specifically, two separate coefficients are introduced to adjust the conditional domain discrepancies of normal instances and outliers with the purpose of alleviating the data imbalance problem. The particle swarm optimizer is employed to adjust the hyper-parameters. The developed deep transfer learning framework is exploited in designing a new outlier detector with application to WAAM. The proposed approach is exploited in real-world industrial data collected through the WAAM process. Experimental results demonstrate that the proposed outlier detection approach outperforms the standard deep-learning-based outlier detector approach and the standard transfer-learning-embedded outlier detection approach in terms of detection accuracy.