Outlier Detection Method Research Articles

Unsupervised Financial Fraud Detection Using Low-rank Recovery

Many outlier detection methods have been studied and applied to financial fraud detection problems in recent years. These traditional financial fraud detection problems (such as classification-based methods) often suffer performance degradation caused by imbalanced data. Even worse, the frauds often destroy the low dimensional structure of the sub-space on which the intrinsic samples are distributed, and the traditional financial fraud detection methods fail to study the low rankness prior in the intrinsic samples when outliers emerge. We give an effective unsupervised financial fraud detection algorithm based on a low-rank recovery method to solve these issues. Our work has several folds: (1) We adopt the Outlier Pursuit (OP) algorithm and its non-convex variants, which can study the low rankness within the intrinsic samples and detect the outliers without supervision. (2) We solve OP and its non-convex variants by combining Alternating Direction Method Of Multipliers (ADMM) with Generalized Accelerating Iterative Algorithm (GAI). (3) We evaluate the effectiveness of the OP-based methods and compare methods on three data sets, including the German Credit Data, Insurance Company (TIC) Benchmark, and IEEE-CIS fraud Detection data set. The related code is available at https://github.com/jzheng20/FinancialFraudDetectionViaOP.git.

Cloud computing English teaching application platform based on machine learning algorithm

Due to the progress of computer technology at this stage, English teaching has gradually achieved reforms with the help of the Internet. This paper builds an English teaching application platform based on machine learning algorithms, and realizes personalized online teaching with the help of cloud computing technology. In view of the complex characteristics of learners’ online learning behavior data, this paper uses Pearson’s correlation coefficient to quantify the correlation between various learning behavior variables. In order to further improve the accuracy of system data analysis, regression analysis is used to detect abnormal data points, and data with a data prediction deviation greater than 1 is determined as abnormal data. In order to reduce the loss of a large amount of data loss caused by the regression analysis to judge some anomalies, this paper uses the local outlier detection method again to detect local outliers, and compares the data repeatedly detected in the two identification algorithms, and judges it as abnormal data special handling. In order to further verify the practicability of the system, network simulation is used to increase the number of concurrent operators in the system. The experiment proves that the click delay time of the English teaching video of the system is maintained below 1.8 s under 5000 concurrent operations, and the running speed of the system is significantly improved when the dynamic scheduling function is enabled, which proves that the system can meet the needs of multiple users for concurrent operations. Finally, a survey is conducted on the personalized education of the English teaching platform, and the relevant experimental and survey data are analyzed in detail, and a series of constructive strategies for the development of personalized English teaching in the cloud education environment are proposed.

Feature selection considering interaction, redundancy and complementarity for outlier detection in categorical data

Feature selection is usually used as a preprocessing step for outlier detection to obtain significant performance. There is little work on feature selection for outlier detection in categorical data, and many studies do not consider the interaction and complementarity of features simultaneously, or even confuse them. In this paper, we first design criteria to measure the feature complementarity and redundancy based on the feature weight gain, and then propose a new feature selection method for outlier detection in categorical data which takes into account the feature relevance, interaction, redundancy and complementarity. The proposed method can find potential complementary features for the selected feature subset containing as much positive information as possible and remove redundant features without beneficial information. Extension evaluation results on 14 real-world datasets demonstrate that the proposed method acquires an average 77% dimensionality reduction rate and enables three distinct types of outlier detection methods to obtain superior AUC than that on the original dataset. Our method performs best in all outlier detectors on average compared with the other five state-of-the-art feature selection methods.

Federated Feature Selection for Horizontal Federated Learning in IoT Networks

Under horizontal federated learning (HFL) in IoT scenarios, different user datasets have significant similarities on the feature spaces, the final goal is to build a high-performance global model. However, not all features are great contributors when training the global HFL model, some features even impair the HFL. Besides, the curse of dimension will delay the training time and cause more energy consumption (EC). In this case, it is critical to remove irrelevant features from the local and select the useful overlapping features from a federated global perspective. In addition, the uncertainty of data being labelled and the non-independent and identically distributed (non-IID) client data should also consider. This paper introduces an unsupervised federated feature selection approach (named FSHFL) for HFL in IoT networks. First, a feature relevance outlier detection method is applied to the HFL participants to remove the useless features, which combines with the improved one-class support vector machine. Besides, a feature relevance hierarchical clustering (FRHC) algorithm is proposed for HFL overlapping feature selection. Experiment results on four IoT datasets show that the proposed methods can select better federated feature sets among HFL participants, thus improving the performance of the HFL system. Specifically, the global model accuracy improves up to 1.68% since fewer irrelevant features. Moreover, FSHFL can lower the average training time as high as 6.9%. Last, when the global model gets the same test accuracy, FSHFL can decrease the average EC of training the model by approximately 2.85% compared to Fed-Avg and roughly 68.39% compared to Fed-SGD.

Identifying Outlier Subjects in Bioavailability Trials using Generalized Studentized Residuals

This paper discusses several outlier detection methods for bioavailability trials, particularly based on residuals. By considering a simplified model of standard crossover model, which is commonly used in bioavailability trials, we propose an outlier detection procedure based on the generalized studentized residuals (SR3) and compare its ability of detecting the possible outlying subjects with two existing procedures, which are carried out based on the classical studentized residual (SR1) and studentized residual using median absolute deviation (SR2). The performances of these procedures in detecting outlying subject are presented via an extensive simulation study. The results show that the proposed procedure SR3 performs more powerful than that using SR1, and as well as the procedure using SR2 for outlier detection. As an illustration, these procedures are implemented on a real dataset from bioavailability study, namely, the area under the curve (AUC) dataset for two erythromycin formulations.

An outlier detection and recovery method based on moving least squares quasi-interpolation scheme and [formula omitted][formula omitted]-minimization problem

In the scattered data used for function approximation, there may be some data that deviate greatly from their true values, which are called outliers. The existence of outliers significantly affects the accuracy of some approximation methods, such as moving least squares. How to quickly and accurately find these outliers and restore their true values has not been well resolved. In this paper, a new outlier detection and recovery method is proposed for data processing. The method uses the high accuracy of moving least squares quasi-interpolation scheme, its sensitivity to outliers and the sparse distribution of outliers to construct a ł0-minimization problem with an inequality constrain. Under certain assumptions, it is proved theoretically that the deviation vector corresponding to the outliers is a solution to the optimization problem. The classical orthogonal matching pursuit algorithm is introduced to solve the optimization problem efficiently. By solving the problem, the outliers are marked, and the deviations are also estimated approximately, which can restore the true values of outliers. The numerical experiments demonstrate that the proposed method has high computational efficiency, very high detection accuracy, and high recovery accuracy for the scattered data used for function approximation, so it is practical.

Preserving microgrid sustainability through robust islanding detection scheme ensuring cyber-situational awareness

Along with the numerous environmental and operational benefits of renewable energy sources (RESs) integration into modern distribution networks, maintaining system security in these networks during challenging phenomena, especially unintentional islanding can still be problematic in some cases. This paper puts forward a sensitivity-based islanding detection algorithm that utilizes the transient kinetic energy of the RESs to identify the islanding condition in microgrids and also considers cyber-attacks, which may lead to operators making inappropriate operational choices. To be specific, the proposed algorithm finds a generalized index that considers simultaneously the impact of volatility of parameters during islanding and non-islanding conditions and is also robust against data integrity attacks. The generalized index, the so-called sensitivity islanding detection (SID) index, is designed based on the sensitivity of transient kinetic energy over transient influential parameters. Furthermore, a statistical index for islanding detection is introduced based on the standard variation that effectively monitors the volatilities. An unsupervised anomaly detection method that uses kernel density estimation technology and statistical calculations are proposed as a way to find cyber-attacks. Following that, an outlier detection method is used to identify anomalies. The effectiveness of the proposed algorithm is evaluated under various conditions using simulation environments.

Outlier Detection with Reinforcement Learning for Costly to Verify Data.

Outliers are often present in data and many algorithms exist to find these outliers. Often we can verify these outliers to determine whether they are data errors or not. Unfortunately, checking such points is time-consuming and the underlying issues leading to the data error can change over time. An outlier detection approach should therefore be able to optimally use the knowledge gained from the verification of the ground truth and adjust accordingly. With advances in machine learning, this can be achieved by applying reinforcement learning on a statistical outlier detection approach. The approach uses an ensemble of proven outlier detection methods in combination with a reinforcement learning approach to tune the coefficients of the ensemble with every additional bit of data. The performance and the applicability of the reinforcement learning outlier detection approach are illustrated using granular data reported by Dutch insurers and pension funds under the Solvency II and FTK frameworks. The application shows that outliers can be identified by the ensemble learner. Moreover, applying the reinforcement learner on top of the ensemble model can further improve the results by optimising the coefficients of the ensemble learner.

A hybrid dimensionality reduction method for outlier detection in high-dimensional data

A hybrid dimensionality reduction method for outlier detection in high-dimensional data

Bivariate logistic regression model diagnostics applied to analysis of outlier cancer patients with comorbid diabetes and hypertension in Malawi

The joint occurrence of diabetes and hypertension conditions in a patient is common. The two diseases share a number of risk factors, and are hence usually modelled concurrently using bivariate logistic regression. However, the postestimation assessment for the model, such as analysis of outlier observations, is seldom carried out. In this article, we apply outlier detection methods for multivariate data models to study characteristics of cancer patients with joint outlying diabetes and hypertension outcomes observed from among 398 randomly selected cancer patients at Queen Elizabeth and Kamuzu Central Hospitals in Malawi. We used R software version 4.2.2 to perform the analyses and STATA version 12 for data cleaning. The results showed that one patient was an outlier to the bivariate diabetes and hypertension logit model. The patient had both diabetes and hypertension and was based in rural area of the study population, where it was observed that comorbidity of the two diseases was uncommon. We recommend thorough analysis of outlier patients to comorbid diabetes and hypertension before rolling out interventions for managing the two diseases in cancer patients to avoid misaligned interventions. Future research could perform the applied diagnostic assessments for the bivariate logit model on a wider and larger dataset of the two diseases.

Extraction and Analysis of Coronal High-temperature Components Based on Outlier Detection

The extraction of high-temperature regions in active regions (ARs) is an important means to help understand the mechanism of coronal heating. The important observational means of high-temperature radiation in ARs is the main emission line of Fe xviii in the 94 Å of the Atmospheric Imaging Assembly. However, the diagnostic algorithms for Fe xviii, including the differential emission measure (DEM) and linear diagnostics proposed by Del based on the DEM, have been greatly limited for a long time, and the results obtained are different from the predictions. In this paper, we use the outlier detection method to establish the nonlinear correlation between 94 Å and 171, 193, 211 Å based on the former researches by others. A neural network based on 171, 193, 211 Å is constructed to replace the low-temperature emission lines in the ARs of 94 Å. The predicted results are regarded as the low-temperature components of 94 Å, and then the predicted results are subtracted from 94 Å to obtain the outlier component of 94 Å, or Fe xviii. Then, the outlier components obtained by neural network are compared with the Fe xviii obtained by DEM and Del’s method, and a high similarity is found, which proves the reliability of neural network to obtain the high-temperature components of ARs, but there are still many differences. In order to analyze the differences between the Fe xviii obtained by the three methods, we subtract the Fe xviii obtained by the DEM and Del’s method from the Fe xviii obtained by the neural network to obtain the residual value, and compare it with the results of Fe xiv in the temperature range of 6.1–6.45 MK. It is found that there is a great similarity, which also shows that the Fe xviii obtained by DEM and Del’s method still has a large low-temperature component dominated by Fe xiv, and the Fe xviii obtained by neural network is relatively pure.

Prediction of some selected soil properties using the Hungarian Mid-infrared spectral library

Routine soil chemical and physical laboratory analysis provides a better understanding of the soil by evaluating its quality and functions. Demands for the development of national Mid-infrared (MIR) spectral libraries for predicting soil attributes with high accuracy have risen substantially in the recent past. Such MIR spectral library is usually regarded as a fast, cheap and non-destructive technique for estimating soil properties compared to laboratory soil analysis. The main objective of this research was to assess the performance of the Hungarian MIR spectral library in estimating four soil properties namely: Cation Exchange Capacity (CEC), Exchangeable Mg and Ca and pH water at different scenarios. Archived soil samples were scanned and spectra data were saved in the Fourier transform infrared spectrometer OPUS software. Preprocessed filtering, outlier detection and calibration sample selection methods were applied to the spectral library. MIR calibration models were built for soil attributes using partial least square regression method and the models were validated with sample predictions. R2, RMSE and RPD were used to assess the goodness of calibration and validation models. MIR spectral library had the ability to estimate soil properties such as CEC and exchangeable Ca and Mg through various scale models (national, county and soil type). The findings showed that the Hungarian MIR spectral library for estimation of soil properties has the ability to provide good information on national, county and soil type scales at different levels of accuracy.

Review of Applicable Outlier Detection Methods to Treat Geomechanical Data

The reliability of geomechanical models and engineering designs depend heavily on high-quality data. In geomechanical projects, collecting and analyzing laboratory data is crucial in characterizing the mechanical properties of soils and rocks. However, insufficient lab data or underestimating data treatment can lead to unreliable data being used in the design stage, causing safety hazards, delays, or failures. Hence, detecting outliers or extreme values is significant for ensuring accurate geomechanical analysis. This study reviews and categorizes applicable outlier detection methods for geomechanical data into fence labeling methods and statistical tests. Using real geomechanical data, the applicability of these methods was examined based on four elements: data distribution, sensitivity to extreme values, sample size, and data skewness. The results indicated that statistical tests were less effective than fence labeling methods in detecting outliers in geomechanical data due to limitations in handling skewed data and small sample sizes. Thus, the best outlier detection method should consider this matter. Fence labeling methods, specifically, the medcouple boxplot and semi-interquartile range rule, were identified as the most accurate outlier detection methods for geomechanical data but may necessitate more advanced statistical techniques. Moreover, Tukey’s boxplot was found unsuitable for geomechanical data due to negative confidence intervals that conflicted with geomechanical principles.

Unsupervised outlier detection for time-series data of indoor air quality using LSTM autoencoder with ensemble method

The proposed framework consists of three modules as an outlier detection method for indoor air quality data. We first use a long short-term memory autoencoder (LSTM-AE) based reconstruction error detector, which designs the LSTM layer in the shape of an autoencoder, to build a reconstruction error-based outlier detection model and extract latent features. The latent feature class-assisted vector machine detector constructs an additional outlier detection model using previously extracted latent features. Finally, the ensemble detector combines the two independent classifiers to define a new ensemble-based decision rule. Furthermore, because real-time anomaly detection proceeds with unsupervised learning, more stable and consistent external detection rules are defined than when using a single ensemble model. Laboratory tests with five random cases were performed for objective evaluation. Thus, we propose a framework that can be applied to various industrial environments by detecting and defining stable outlier decision rules.

On the evaluation of outlier detection and one-class classification: a comparative study of algorithms, model selection, and ensembles

It has been shown that unsupervised outlier detection methods can be adapted to the one-class classification problem (Janssens and Postma, in: Proceedings of the 18th annual Belgian-Dutch on machine learning, pp 56–64, 2009; Janssens et al. in: Proceedings of the 2009 ICMLA international conference on machine learning and applications, IEEE Computer Society, pp 147–153, 2009. https://doi.org/10.1109/ICMLA.2009.16). In this paper, we focus on the comparison of one-class classification algorithms with such adapted unsupervised outlier detection methods, improving on previous comparison studies in several important aspects. We study a number of one-class classification and unsupervised outlier detection methods in a rigorous experimental setup, comparing them on a large number of datasets with different characteristics, using different performance measures. In contrast to previous comparison studies, where the models (algorithms, parameters) are selected by using examples from both classes (outlier and inlier), here we also study and compare different approaches for model selection in the absence of examples from the outlier class, which is more realistic for practical applications since labeled outliers are rarely available. Our results showed that, overall, SVDD and GMM are top-performers, regardless of whether the ground truth is used for parameter selection or not. However, in specific application scenarios, other methods exhibited better performance. Combining one-class classifiers into ensembles showed better performance than individual methods in terms of accuracy, as long as the ensemble members are properly selected.

Streak artifact suppressed back projection for sparse-view photoacoustic computed tomography.

The development of fast and accurate image reconstruction algorithms under constrained data acquisition conditions is important for photoacoustic computed tomography (PACT). Sparse-view measurements have been used to accelerate data acquisition and reduce system complexity; however, reconstructed images suffer from sparsity-induced streak artifacts. In this paper, a modified back-projection (BP) method termed anti-streak BP is proposed to suppress streak artifacts in sparse-view PACT reconstruction. During the reconstruction process, the anti-streak BP finds the back-projection terms contaminated by high-intensity sources with an outlier detection method. Then, the weights of the contaminated back-projection terms are adaptively adjusted to eliminate the effects of high-intensity sources. The proposed anti-streak BP method is compared with the conventional BP method on both simulation and in vivo data. The anti-streak BP method shows substantially fewer artifacts in the reconstructed images, and the streak index is 54% and 20% lower than that of the conventional BP method on simulation and in vivo data, when the transducer number N=128. The anti-streak BP method is a powerful improvement of the BP method with the ability of artifact suppression.

Short tandem repeat expansions in sporadic amyotrophic lateral sclerosis and frontotemporal dementia

Pathogenic short tandem repeat (STR) expansions cause over 20 neurodegenerative diseases. To determine the contribution of STRs in sporadic amyotrophic lateral sclerosis (ALS) and frontotemporal dementia (FTD), we used ExpansionHunter, REviewer, and polymerase chain reaction validation to assess 21 neurodegenerative disease-associated STRs in whole-genome sequencing data from 608 patients with sporadic ALS, 68 patients with sporadic FTD, and 4703 matched controls. We also propose a data-derived outlier detection method for defining allele thresholds in rare STRs. Excluding C9orf72 repeat expansions, 17.6% of clinically diagnosed ALS and FTD cases had at least one expanded STR allele reported to be pathogenic or intermediate for another neurodegenerative disease. We identified and validated 162 disease-relevant STR expansions in C9orf72 (ALS/FTD), ATXN1 [spinal cerebellar ataxia type 1 (SCA1)], ATXN2 (SCA2), ATXN8 (SCA8), TBP (SCA17), HTT (Huntington's disease), DMPK [myotonic dystrophy type 1 (DM1)], CNBP (DM2), and FMR1 (fragile-X disorders). Our findings suggest clinical and pathological pleiotropy of neurodegenerative disease genes and highlight their importance in ALS and FTD.

Evaluation of Unsupervised Anomaly Detection Techniques in Labelling Epileptic Seizures on Human EEG

Automated labelling of epileptic seizures on electroencephalograms is an essential interdisciplinary task of diagnostics. Traditional machine learning approaches operate in a supervised fashion requiring complex pre-processing procedures that are usually labour intensive and time-consuming. The biggest issue with the analysis of electroencephalograms is the artefacts caused by head movements, eye blinks, and other non-physiological reasons. Similarly to epileptic seizures, artefacts produce rare high-amplitude spikes on electroencephalograms, complicating their separability. We suggest that artefacts and seizures are rare events; therefore, separating them from the rest data seriously reduces information for further processing. Based on the occasional nature of these events and their distinctive pattern, we propose using anomaly detection algorithms for their detection. These algorithms are unsupervised and require minimal pre-processing. In this work, we test the possibility of an anomaly (or outlier) detection algorithm to detect seizures. We compared the state-of-the-art outlier detection algorithms and showed how their performance varied depending on input data. Our results evidence that outlier detection methods can detect all seizures reaching 100% recall, while their precision barely exceeds 30%. However, the small number of seizures means that the algorithm outputs a set of few events that could be quickly classified by an expert. Thus, we believe that outlier detection algorithms could be used for the rapid analysis of electroencephalograms to save the time and effort of experts.

Outlier Detection Based on Nelder-Mead Simplex Robust Kalman Filtering for Trustworthy Bridge Structural Health Monitoring

Structural health monitoring (SHM) is vital for ensuring the service safety of aging bridges. As one of the most advanced sensing techniques, Global Navigation Satellite Systems (GNSS) could capture massive spatiotemporal information for effective bridge structural health monitoring (BSHM). Unfortunately, GNSS measurements often contain outliers due to various factors (e.g., severe weather conditions, multipath effects, etc.). All such outliers could jeopardize the accuracy and reliability of BSHM significantly. Previous studies have examined the feasibility of integrating the conventional multi-rate Kalman filter (MKF) with an adaptive algorithm in the data processing processes to ensure BSHM accuracy. However, frequent parameter adjustments are still needed in tedious data processing processes. This study proposed an outlier detection method using a Nelder-Mead simplex robust multi-rate Kalman filter (RMKF) for supporting trustworthy BSHM using GNSS and accelerometer. In the end, the authors have validated the proposed method using the monitoring data collected at the Wilford Bridge in the UK. Results showed that the accuracy of the total dynamic vibration displacement time series has been improved by 21% compared with the results using the conventional MKF approach. The authors envision that the proposed method will shed light on reliable and explainable data processing policy and trustworthy BSHM.

A Novel Model-Based Reinforcement Learning for Online Anomaly Detection in Smart Power Grid

Smart grids must detect cyber-attacks early to ensure their safety and reliability. There have been many outlier detection methods presented in the studies, varying from those requiring instance-by-instance decisions t the online diagnosing methods that require the use of accurate models of an attack. This study proposes a novel intelligent online anomaly or attack detection method based on the partially observable Markov decision procedure (POMDP). The proposed model may be categorized as a general detection method according to the reinforcement learning (RL) architecture for POMDP which can help the learning process based on the award concept. The performance of the proposed model is verified using the IEEE test system. Based on numerical results, the suggested RL-based algorithm shows to be very effective in detecting cyber-attacks against the smart grid quickly and accurately.