Outlier Detection Method Research Articles

Prediction of Potato (Solanum tuberosum L.) Yield Based on Machine Learning Methods

This research delves into the application of machine learning methods for predicting the yield of potato varieties used for French fries in Poland. By integrating a comprehensive dataset comprising agronomical, climatic, soil, and satellite-based vegetation data from 36 commercial potato fields over five growing seasons (2018–2022), we developed three distinct models: non-satellite, satellite, and hybrid. The non-satellite model, relying on 85 features, excludes vegetation indices, whereas the satellite model includes these indices within its 128 features. The hybrid model, combining all available features, encompasses a total of 165 features, presenting the most-comprehensive approach. Our findings revealed that the hybrid model, particularly when enhanced with SVM outlier detection, exhibited superior performance with the lowest Mean Absolute Percentage Error (MAPE) of 5.85%, underscoring the effectiveness of integrating diverse data sources into agricultural yield prediction. In contrast, the non-satellite and satellite models displayed higher MAPE values, indicating less accuracy compared to the hybrid model. Advanced data-processing techniques such as PCA and outlier detection methods (LOF and One-Class SVM) played a pivotal role in model performance, optimising feature selection and dataset refinement. The study concluded that machine learning methods, particularly when leveraging a multifaceted approach involving a wide array of data sources and advanced processing techniques, can significantly enhance the accuracy of agricultural yield predictions. These insights pave the way for more-efficient and -informed agricultural practices, emphasising the potential of machine learning in revolutionising yield prediction and crop management.

A Novel Wind Power Outlier Detection Method with Support Vector Machine Optimized by Improved Harris Hawk

The accurate detection of wind power outliers plays a crucial role in wind power forecasting, while the inherited strong randomness and high fluctuations bring great challenges to this issue. This work investigates the way to improve the outlier detection accuracy based on support vector machine (SVM). Although SVM can achieve good results for outlier detection in theory, its performance is heavily dependent on the hyper-parameters. Parameter optimization is not an easy task due to its complex nonlinear multi-optimum nature; an improved Harris hawk optimization (IHHO) is proposed to optimize the parameters of SVM for more accurate outlier detection. HHO takes the cooperative behavior and chasing style of Harris’ hawks in nature called surprise pounce and can effectively search the optimal one in large parameter space, but it tends to fall into local optimum. To solve this issue, an improved Harris hawk optimization algorithm (IHHO) was proposed to obtain the optimal parameters of SVM. First, Hammersley sequence initialization is carried out to acquire good initial solutions. Then, a nonlinear factor control mode and an adaptive Gaussian–Cauchy mutation perturbation strategy are proposed to avoid getting trapped in local optima. In this way, a novel wind power outlier detection method named IHHO-SVM was constructed. The results on several wind power data with outliers show that IHHO-SVM outperforms SVM and HHO-SVM, which achieves the highest average F1 score of 96.63% and exhibits the smallest standard deviation. Compared to commonly used models for detecting outliers in wind power, such as isolation forest (IF), local outlier factor (LOF), SVM with grey wolf optimization (GWO-SVM), and SVM with particle swarm optimization (PSO-SVM), the proposed IHHO-SVM model shows the best overall performance with precision, recall, and F1 scores of 95.76%, 96.94%, and 96.35%, respectively.

A COMPARISON OF PRE-PROCESSING APPROACHES FOR REMOTELY SENSED TIME SERIES CLASSIFICATION BASED ON FUNCTIONAL ANALYSIS

Abstract. Satellite remote sensing has gained a key role for vegetation mapping distribution. Given the availability of multi-temporal satellite data, seasonal variations in vegetation dynamics can be used trough time series analysis for vegetation distribution mapping. These types of data have a very high variability within them and are subjected by artifacts. Therefore, a pre-processing phase must be performed to properly detect outliers, for data smoothing process and to correctly interpolate the data. In this work, we compare four pre-processing approaches for functional analysis on 4-years of remotely sensed images, resulting in four time series datasets. The methodologies presented are the results of the combination of two outlier detection methods, namely tsclean and boxplot functions in R and two discrete data smoothing approaches (Generalized Additive Model ”GAM” on daily and aggregated data). The approaches proposed are: tsclean-GAM on aggregated data (M01), boxplot-GAM on aggregated data (M02), tsclean-GAM on daily data (M03), boxplot-GAM on daily data (M04). Our results prove that the approach which involves tsclean function and GAM applied to daily data (M03) is ameliorative to the logic of the procedure and leads to better model performance in terms of Overall Accuracy (OA) which is always among the highest when compared with the others obtained from the other three different approaches.

A Novel Unsupervised Outlier Detection Algorithm Based on Mutual Information and Reduced Spectral Clustering

Outlier detection is an essential research field in data mining, especially in the areas of network security, credit card fraud detection, industrial flaw detection, etc. The existing outlier detection algorithms, which can be divided into supervised methods and unsupervised methods, suffer from the following problems: curse of dimensionality, lack of labeled data, and hyperparameter tuning. To address these issues, we present a novel unsupervised outlier detection algorithm based on mutual information and reduced spectral clustering, called MISC-OD (Mutual Information and reduced Spectral Clustering—Outlier Detection). MISC-OD first constructs a mutual information matrix between features, then, by applying reduced spectral clustering, divides the feature set into subsets, utilizing the LOF (Local Outlier Factor) for outlier detection within each subset and combining the outlier scores found within each subset. Finally, it outputs the outlier score. Our contributions are as follows: (1) we propose a novel outlier detection method called MISC-OD with high interpretability and scalability; (2) numerous experiments on 18 benchmark datasets demonstrate the superior performance of the MISC-OD algorithm compared with eight state-of-the-art baselines in terms of ROC (receiver operating characteristic) and AP (average precision).

ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions

Outlier detection refers to the identification of data points that deviate from a general data distribution. Existing unsupervised approaches often suffer from high computational cost, complex hyperparameter tuning, and limited interpretability, especially when working with large, high-dimensional datasets. To address these issues, we present a simple yet effective algorithm called ECOD (Empirical-Cumulative-distribution-based Outlier Detection), which is inspired by the fact that outliers are often the "rare events" that appear in the tails of a distribution. In a nutshell, ECOD first estimates the underlying distribution of the input data in a nonparametric fashion by computing the empirical cumulative distribution per dimension of the data. ECOD then uses these empirical distributions to estimate tail probabilities per dimension for each data point. Finally, ECOD computes an outlier score of each data point by aggregating estimated tail probabilities across dimensions. Our contributions are as follows: (1) we propose a novel outlier detection method called ECOD, which is both parameter-free and easy to interpret; (2) we perform extensive experiments on 30 benchmark datasets, where we find that ECOD outperforms 11 state-of-the-art baselines in terms of accuracy, efficiency, and scalability; and (3) we release an easy-to-use and scalable (with distributed support) Python implementation for accessibility and reproducibility.

Enhancing anomaly detectors with LatentOut

LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} is a recently introduced algorithm for unsupervised anomaly detection which enhances latent space-based neural methods, namely (Variational) Autoencoders, GANomaly and ANOGan architectures. The main idea behind it is to exploit both the latent space and the baseline score of these architectures in order to provide a refined anomaly score performing density estimation in the augmented latent-space/baseline-score feature space. In this paper we investigate the performance of LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} acting as a one-class classifier and we experiment the combination of LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} with GAAL architectures, a novel type of Generative Adversarial Networks for unsupervised anomaly detection. Moreover, we show that the feature space induced by LatentOut\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\ extbf{Latent}}\\varvec{Out}}$$\\end{document} has the characteristic to enhance the separation between normal and anomalous data. Indeed, we prove that standard data mining outlier detection methods perform better when applied on this novel augmented latent space rather than on the original data space.

Using eigenvalues of distance matrices for outlier detection

Distance or dissimilarity matrices are widely used in applications. We study the relationships between the eigenvalues of the distance matrices and outliers and show that outliers affect the pairwise distances and inflate the eigenvalues. We obtain the eigenvalues of a distance matrix that is affected by k outliers and compare them to the eigenvalues of a distance matrix with a constant structure. We show a discrepancy in the sizes of the eigenvalues of a distance matrix that is contaminated with outliers, present an algorithm and offer a new outlier detection method based on the eigenvalues of the distance matrix. We compare the new distance-based outlier technique with several existing methods under five distributions. The methods are applied to a study of public utility companies and gene expression data.

Outlier detection for partially labeled categorical data based on conditional information entropy

Labeling a large amount of data is exceptionally costly and practically infeasible, and thus available data may have missing labels. In this article, we investigate outlier detection for partially labeled categorical data based on conditional information entropy. Firstly, the equivalence class in a partially labeled categorical decision information system (p-CDIS) is introduced, so that the missing labels can be predicted by use of conditional probability. Then, conditional information entropy in a p-CDIS is calculated, which provides a more comprehensive measure of uncertainty. Additionally, the relative information entropy and relative cardinality in a p-CDIS are proposed. Next, the degree of outlierness and the weight function are presented to find outlier factors. Finally, an outlier detection method in a p-CDIS based on conditional information entropy is proposed, and a corresponding conditional information entropy algorithm (CEOF) is designed. To evaluate the stability of the CEOF algorithm, experiments are performed on ten UCI Machine Learning Repository datasets. Compared with five other algorithms, the proposed method is shown to have good effectiveness and adaptability for categorical data.

Comparison of Classification Models for Breast Cancer Disease Using Multivariate Analysis and Data Mining Approaches

Compared to other cancer types, breast cancer is one of the main causes of death in women. Early cancer detection can significantly increase survival and quality of life. A variety of machine learning prediction algorithms with combination of feature selection approaches have shown to be useful in the detection of breast cancer disease. However, it was discovered that there are still problems with classification accuracy. An outlier-related factor was known to have potential effect on classification accuracy. In order to further improve the classification’s accuracy, the Kmeans approach was used to detect outliers. The major goal of this study was to examine the classification performance of breast cancer disease when feature selection methods were used in combination with K-Means. For experimental purpose, the Coimbra dataset for breast cancer consisting of 116 instances and 10 attributes was used in this study. Multivariate techniques including Principal Component Analysis (PCA), Kernel Principal Component Analysis (KPCA), and Discriminant Analysis (DA) were applied to reduce data dimensions. Meanwhile, four data mining approaches consisting of Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), and Logistic Regression (LR) were compared for classification purpose. The performance measurement was then evaluated using accuracy, precision, specificity, and sensitivity criteria. The results revealed that five combinations approaches (PCA-DT, PCA-RF, KPCA-DT, KPCA-RF, DA-RF) performed better across all four criteria after combining with KMeans technique. Among five combined methods, KPCA with DT outperformed other combination methods with the highest value across precision (76.47 percent) and specificity (71.43 percent). This study suggests the incorporation of feature selection method together with outlier detection method has proved to be more efficient and beneficial for breast cancer classification.

A KRR-UKF robust state estimation method for distribution networks

State estimation is an integral component of energy management systems. Employing a state estimation methodology that is both accurate and resilient is essential for facilitating informed decision-making processes. However, the complex scenarios (unknown noise, low data redundancy, and reconfiguration) of the distribution network pose new challenges for state estimation. In the context of this study, we introduce a state estimation technique known as the kernel ridge regression and unscented Kalman filter. In normal conditions, the non-linear correlation among data and unknown noise increases the difficulty of modeling the distribution network. Thence, kernel ridge regression is developed to map the data into high-dimensional space that transforms the non-linear problem into linear formulations base on the data rather the complicate grid model, which improves model generalization performance and filters out unknown noises. In addition, with the unique prediction correction mechanism of the Kalman method, the kernel ridge regression-mapped state value can be revised by the measurement, which further enhances model accuracy and robustness. During abnormal operating conditions and taking into account the presence of faulty data within the measurement system, we initiate the use of a long short-term memory network and combined convolutional neural network (CNN) model, referred to as the ATT-CNN-GRU. This model is utilized for the prediction of pseudo-measurements. Subsequently, we use an outlier detection method known as ordering points to identify the clustering structure to effectively identify and substitute erroneous data points. Cases on the IEEE-33 bus system and 109-bus system from a city in China show that the method has superior accuracy and robustness.

Robust Fusion of Multi-Source Images for Accurate 3D Reconstruction of Complex Urban Scenes

Integrated reconstruction is crucial for 3D modeling urban scenes using multi-source images. However, large viewpoint and illumination variations pose challenges to existing solutions. A novel approach for accurate 3D reconstruction of complex urban scenes based on robust fusion of multi-source images is proposed. Firstly, georeferenced sparse models are reconstructed from the terrestrial and aerial images using GNSS-aided incremental SfM, respectively. Then, cross-platform match pairs are selected based on point-on-image observability. The terrestrial and aerial images are robustly matched based on the selected match pairs to generate cross-platform tie points. Thirdly, the tie points are triangulated to derive cross-platform 3D correspondences. The 3D correspondences are refined using a novel outlier detection method. Finally, the terrestrial and aerial sparse models are merged based on the refined correspondences, and the integrated model is globally optimized to obtain an accurate reconstruction of the scene. The proposed methodology is evaluated on five benchmark datasets, and extensive experiments are performed. The proposed pipeline is compared with a state-of-the-art methodology and three widely used software packages. Experimental results demonstrate that the proposed methodology outperforms the other pipelines in terms of robustness and accuracy.

Fusing multi-scale fuzzy information to detect outliers

Outlier detection aims to find objects that behave differently from the majority of the data. Existing unsupervised approaches often process data with a single scale, which may not capture the multi-scale nature of the data. In this paper, we propose a novel information fusion model based on multi-scale fuzzy granules and an unsupervised outlier detection algorithm with the fuzzy rough set theory. First, a multi-scale information fusion model is formulated based on fuzzy granules. Then we employ fuzzy approximations to define the outlier factor of multi-scale fuzzy granules centered at each data point. Finally, the outlier score is calculated by aggregating the outlier factors of a set of multi-scale fuzzy granules. Experimental results demonstrate that the proposed method is comparable with or better than the leading outlier detection methods. The codes and datasets are publicly available online at https://github.com/ChenBaiyang/MFIOD.

Neighborhood Rough Residual Network–Based Outlier Detection Method in IoT-Enabled Maritime Transportation Systems

Outlier detection can identify anomalies in large-scale data. To provide reliability and security for Internet of Things (IoT)-enabled maritime transportation systems (MTSs), in this paper we propose an outlier detection method based on the neighborhood rough residual network (NRRN). We calculate the neighborhood approximation accuracy and neighborhood conditional entropy to obtain the neighborhood combined entropy describing the discrimination ability of the condition attribute subset to the information system. We then delete the redundant attributes according to the attribute combination importance derived from the neighborhood combined entropy. The data after attribute reduction are used to train the convolutional neural network, and the residual network (ResNet50) is used to avoid the degradation of model performance caused by the increase in the number of network layers. The proposed method is compared with mainstream outlier detection algorithms on a fishing vessel operation dataset. Experiments show that the proposed method can greatly improve the accuracy of outlier detection while taking into account interpretability and computational efficiency, thereby ensuring the data integrity of IoT-enabled MTSs.

Detecting unexpected growths in health technologies expenditures: the case of MIPRES in Colombia

We developed an algorithm to explore unexpected growth in the usage and costs of health technologies. We exploit data from the expenditures on technologies funded by the Colombian government under the compulsory insurance system, where all prescriptions for technologies not included in an explicit list must be registered in a centralized information system, covering the period from 2017 to 2022. The algorithm consists of two steps: an outlier detection method based on the density of the expenditures for selecting a first set of technologies to consider (39 technologies out of 106,957), and two anomaly detection models for time series to determine which insurance companies, health providers, and regions have the most notorious increases. We have found that most medicines associated with atypical behavior and significant monetary growth could be linked to the use of recently introduced drugs in the market. These drugs have valid patents and very specific clinical indications, often involving high-cost pharmacological treatments. The most relevant case is the Burosumab, approved in 2018 to treat a rare genetic disorder affecting skeletal growth. Secondly, there is clear evidence of anomalous increasing trend evolutions in the identified enteral nutritional support supplements or Food for Special Medical Purposes. The health system did not purchase these products before July 2021, but in 2022 they represented more than 500,000 USD per month.

A data-adaptive method for outlier detection from functional data

A data-adaptive method for outlier detection from functional data

New approaches and technical considerations in detecting outlier measurements and trajectories in longitudinal children growth data

BackgroundGrowth studies rely on longitudinal measurements, typically represented as trajectories. However, anthropometry is prone to errors that can generate outliers. While various methods are available for detecting outlier measurements, a gold standard has yet to be identified, and there is no established method for outlying trajectories. Thus, outlier types and their effects on growth pattern detection still need to be investigated. This work aimed to assess the performance of six methods at detecting different types of outliers, propose two novel methods for outlier trajectory detection and evaluate how outliers affect growth pattern detection.MethodsWe included 393 healthy infants from The Applied Research Group for Kids (TARGet Kids!) cohort and 1651 children with severe malnutrition from the co-trimoxazole prophylaxis clinical trial. We injected outliers of three types and six intensities and applied four outlier detection methods for measurements (model-based and World Health Organization cut-offs-based) and two for trajectories. We also assessed growth pattern detection before and after outlier injection using time series clustering and latent class mixed models. Error type, intensity, and population affected method performance.ResultsModel-based outlier detection methods performed best for measurements with precision between 5.72-99.89%, especially for low and moderate error intensities. The clustering-based outlier trajectory method had high precision of 14.93-99.12%. Combining methods improved the detection rate to 21.82% in outlier measurements. Finally, when comparing growth groups with and without outliers, the outliers were shown to alter group membership by 57.9 -79.04%.ConclusionsWorld Health Organization cut-off-based techniques were shown to perform well in few very particular cases (extreme errors of high intensity), while model-based techniques performed well, especially for moderate errors of low intensity. Clustering-based outlier trajectory detection performed exceptionally well across all types and intensities of errors, indicating a potential strategic change in how outliers in growth data are viewed. Finally, the importance of detecting outliers was shown, given its impact on children growth studies, as demonstrated by comparing results of growth group detection.

Outlier detection using iterative adaptive mini-minimum spanning tree generation with applications on medical data.

As an important technique for data pre-processing, outlier detection plays a crucial role in various real applications and has gained substantial attention, especially in medical fields. Despite the importance of outlier detection, many existing methods are vulnerable to the distribution of outliers and require prior knowledge, such as the outlier proportion. To address this problem to some extent, this article proposes an adaptive mini-minimum spanning tree-based outlier detection (MMOD) method, which utilizes a novel distance measure by scaling the Euclidean distance. For datasets containing different densities and taking on different shapes, our method can identify outliers without prior knowledge of outlier percentages. The results on both real-world medical data corpora and intuitive synthetic datasets demonstrate the effectiveness of the proposed method compared to state-of-the-art methods.

Outlier detection for incomplete real-valued data via rough set theory and granular computing

Outlier detection is an important topic in data mining. An information system (IS) is a database that shows relationships between objects and attributes. A real-valued information system (RVIS) is an IS whose information values are real numbers. People often encounter missing values during data processing. A RVIS with the miss values is an incomplete real-valued information system (IRVIS). Due to the presence of the missing values, the distance between two information values is difficult to determine, so the existing outlier detection rarely considered an IS with the miss values. This paper investigates outlier detection for an IRVIS via rough set theory and granular computing. Firstly, the distance between two information values on each attribute of an IRVIS is introduced, and the parameter λ to control the distance is given. Then, the tolerance relation on the object set is defined according to the distance, and the tolerance class is obtained, which is regarded as an information granule. After then, λ-lower and λ-upper approximations in an IRVIS are put forward. Next, the outlier factor of every object in an IRVIS is presented. Finally, outlier detection method for IRVIS via rough set theory and granular computing is proposed, and the corresponding algorithms is designed. Through the experiments, the proposed method is compared with other methods. The experimental results show that the designed algorithm is more effective than some existing algorithms in an IRVIS. It is worth mentioning that for comprehensive comparison, ROC curve and AUC value are used to illustrate the advantages of the proposed method.

Human Activity Recognition with Smartwatch Data by using Mahalanobis Distance-Based Outlier Detection and Ensemble Learning Methods

Recognition of human activities is part of smart healthcare applications. In this context, the detection of human actions with high accuracy has been a field that has been working for many years. With the increase in the usage of smart devices, smartphones and smartwatches have become the constant equipment of these studies thanks to their internal sensors. Sometimes abnormal data are included in data sets due to the way the data were collected and for reasons arising from the sensors. For this reason, it becomes important to detect outlier data. In this study, step counter and heart rate sensors were used in addition to an accelerometer and gyroscope in order to detect human activities. Afterward, the outliers were detected and cleared with a Mahalanobis distance-based approach. With the aim of achieving a better classification performance, machine learning methods were used by strengthening them with ensemble learning methods. The obtained results showed that step counter, heart rate sensors, and ensemble learning methods positively affect the success of the classification. In addition, it was found that the Mahalanobis distance-based outlier detection method increased the classification accuracy significantly.

A novel subspace outlier detection method by entropy-based clustering algorithm

Subspace outlier detection has emerged as a practical approach for outlier detection. Classical full space outlier detection methods become ineffective in high dimensional data due to the “curse of dimensionality”. Subspace outlier detection methods have great potential to overcome the problem. However, the challenge becomes how to determine which subspaces to be used for outlier detection among a huge number of all subspaces. In this paper, firstly, we propose an intuitive definition of outliers in subspaces. We study the desirable properties of subspaces for outlier detection and investigate the metrics for those properties. Then, a novel subspace outlier detection algorithm with a statistical foundation is proposed. Our method selectively leverages a limited set of the most interesting subspaces for outlier detection. Through experimental validation, we demonstrate that identifying outliers within this reduced set of highly interesting subspaces yields significantly higher accuracy compared to analyzing the entire feature space. We show by experiments that the proposed method outperforms competing subspace outlier detection approaches on real world data sets.