Outlier Detection Approach Research Articles

Simple and effective approach for construction of universal blind steganalyzer

The goal of universal blind steganalysis is to detect all known (already existing) and unknown (previously unseen) steganographic algorithms without knowledge of the exact stego algorithm used by the steganographer. However, a binary blind steganalyzer trained on cover images and stego images randomly selected from “known stego images” (i.e., stego images produced by multiple “known” stego methods with a mixture of payloads), may fail catastrophically on unknown stego methods although shows superior performance on known stego methods. Additionally, unsupervised outlier detection and one-class classification approaches are less likely to fail to detect unknown stego methods but yield high false positive rates. Motivated by these observations, we explore a simple and effective approach for construction of universal blind steganalyzer to achieve overall good performance on both known and unknown stego algorithms. First, we compute Local Outlier Factor (LOF) scores of known stego sample points (feature vectors) with respect to test sample points. Then, we choose stego images with the lowest LOF scores from known stego images as training stego images. Finally, we train a binary classifier on cover images and chosen training stego images for test. Experimental results confirm that the proposed approach performs significantly better than the random sampling-based binary classification method, unsupervised outlier detection and one-class classification approaches on both known and unknown stego algorithms.

A novel approach for graph-based global outlier detection in social networks

Graph representation has high expensive power to model and detect complicated structural patterns. One important area of data mining that uses such representation is anomaly detection, particularly in the social network graph to ensure network privacy, and uncover interesting behaviour. In this work, we suggest a new approach for global outlier detection in social networks based on graph pattern matching. A node signature extraction is combined with an optimal assignment method for matching the original graph data with the graph pattern data, in order to detect two formalised anomalies: anomalous nodes and anomalous edges. First, we introduce Euclidean and Gower formulas to compute the distance between graphs. Then, we conduct graph pattern matching in cubic-time by defining a node-to-node cost in an assignment problem using the Hungarian method. Finally, the obtained experimental results demonstrate that our approach performs on both synthetic and real social network datasets.

A novel approach for graph-based global outlier detection in social networks

Graph representation has high expensive power to model and detect complicated structural patterns. One important area of data mining that uses such representation is anomaly detection, particularly in the social network graph to ensure network privacy, and uncover interesting behaviour. In this work, we suggest a new approach for global outlier detection in social networks based on graph pattern matching. A node signature extraction is combined with an optimal assignment method for matching the original graph data with the graph pattern data, in order to detect two formalised anomalies: anomalous nodes and anomalous edges. First, we introduce Euclidean and Gower formulas to compute the distance between graphs. Then, we conduct graph pattern matching in cubic-time by defining a node-to-node cost in an assignment problem using the Hungarian method. Finally, the obtained experimental results demonstrate that our approach performs on both synthetic and real social network datasets.

Comparative Analysis Of K-Means Data Mining and Outlier Detection Approach for Network-Based Intrusion Detection

New kind of intrusions causes deviation in the normal behaviour of traffic flow in computer networks every day. This study focused on enhancing the learning capabilities of IDS to detect the anomalies present in a network traffic flow by comparing the k-means approach of data mining for intrusion detection and the outlier detection approach. The k-means approach uses clustering mechanisms to group the traffic flow data into normal and abnormal clusters. Outlier detection calculates an outlier score (neighbourhood outlier factor NOF) for each flow record, whose value decides whether a traffic flow is normal or abnormal. These two methods were then compared in terms of various performance metrics and the amount of computer resources consumed by them. Overall, k-means was more accurate and precise and has better classification rate than outlier detection in intrusion detection using traffic flows. This will help systems administrators in their choice of IDS.

Učinkovitost različnih načinov odkrivanja grobih pogreškov v geodetskih mrežah: primer spomenika Pobednik

Učinkovitost različnih načinov odkrivanja grobih pogreškov v geodetskih mrežah: primer spomenika Pobednik

Community outlier detection in social networks based on graph matching

Outlier or anomaly detection is widely used in several fields of study such as statistics, data mining, and social networks. It can reveal important anomalous and interesting outlier behaviours in the social network communities. In this work, we propose a new approach for community outlier detection based on social network graph matching. We identify community structures in social networks using some community detection methods. For each community, the node signature is combined with an optimal assignment method for matching the original graph data with the graph pattern data, in order to detect two formalised anomalies: anomalous nodes and anomalous edges. We also define a distance between two graphs using Euclidean formula. Then, we define a node-to-node cost in an assignment problem using the Hungarian method to deduce the matching function. The obtained experimental results demonstrate that our approach performs on real social network datasets.

Community outlier detection in social networks based on graph matching

Outlier or anomaly detection is widely used in several fields of study such as statistics, data mining, and social networks. It can reveal important anomalous and interesting outlier behaviours in the social network communities. In this work, we propose a new approach for community outlier detection based on social network graph matching. We identify community structures in social networks using some community detection methods. For each community, the node signature is combined with an optimal assignment method for matching the original graph data with the graph pattern data, in order to detect two formalised anomalies: anomalous nodes and anomalous edges. We also define a distance between two graphs using Euclidean formula. Then, we define a node-to-node cost in an assignment problem using the Hungarian method to deduce the matching function. The obtained experimental results demonstrate that our approach performs on real social network datasets.

Short-Term Solar Power Forecasting Based on Weighted Gaussian Process Regression

Photovoltaic (PV) power is volatile in nature and raises the level of uncertainty in power systems. PV power forecasting is an important measure to solve this problem. It helps to improve the reliability and reduces the generation cost. Advances in computer technology and sensors make the numeric modeling methods a hotspot in the field of PV power forecasting. However, data modeling methods strongly rely on the accuracy of measurement data. Unavoidable outliers in the measured meteorological data have an adverse effect on the model due to their heteroscedasticity. Although many studies can be found focusing on outlier detection, only a few have incorporated outlier detection with regression models. In this study, an innovative method employing the weighted Gaussian process regression approach is proposed, such that data samples with higher outlier potential have a low weight. A density-based local outlier detection approach is introduced to compensate the deterioration of Euclidean distance for high-dimensional data. A novel concept of the degree of nonlinear correlation is incorporated to compute the contribution of every individual data attribute. Effectiveness of the proposed method is demonstrated by performing an experimental analysis and making comparisons with other typical data-based approaches, and the results exhibit higher estimation accuracy.

Protein-Protein Etkileşim Verileri için Aykırı Değer Tespiti Uygulanmasi Gerekir mi?

Objective: Outlier detection is a crucial problem in many fields. Although there are too many outlier detection methods in the literature, only a few methods suitable for dependent, sparse and high dimensional data structure. In this study, we perform various univariate and multivariate outlier detection methods as a pre-processing step before modeling the protein-protein interaction networks in order to investigate whether the outlier detection can improve the accuracy of the model. Material and Methods: Within the univariate approaches, we implement the z-score and Box-plot methods which are the most well-known outlier detection approaches. Besides them, we also apply the multivariate outlier detection methods, called PCOut and Sign which are based on the robust principal component analysis and the BACON method which is a distance-based approach. These methods are applicable for the data type such that the number of variables are bigger than the observation. In the analysis, we use several synthetic and real benchmark biological datasets. Then, we infer the networks with 3 network models, namely, GGM, MARS and CMARS and finally, we check the validity of models via F-measure and the accuracy measures. Conclusion: From the results, it has been seen that the use of outlier detection methods before the modeling cannot contribute to the performance of the models in our datasets. Results: Based on the results obtained from different datasets, we suggest that the estimations of protein-protein interaction networks can be made with GGM, MARS and CMARS methods without the need for an outlier detection process.

Trajectory outlier detection approach based on common slices sub-sequence

Trajectory outlier detection is one of the most popular trajectory data mining topics. It helps researchers obtain a lot of valuable information that can be used as important guidance in monitoring and forecasting. Existing methods have difficulty in detecting the outlying trajectories with continuous multi-segment exception. To address the problem, in this paper, we propose a novel trajectory outlier detection algorithm based on common slices sub-sequence (TODCSS). For each trajectory, the direction-code sequence is firstly calculated based on the direction of each trajectory segment. Secondly, the corresponding sequence consisting of trajectory slices is obtained by inflection point segmentation. And then, the common slices sub-sequences between two trajectories are found to measure their distance. Finally, the slice outliers and trajectory outliers are detected based on the new CSS distance calculation. Both the intuitive visualization presentation and the experimental results on real Atlantic hurricane dataset, real-life mobility trajectory dataset of taxis in San Francisco and synthetic labeled dataset show that the proposed TODCSS algorithm effectively detects slice and trajectory outliers, and improves accuracy and stability in trajectory outlier detection.

Outlier detection using difference-based variance estimators in multiple regression

ABSTRACTIn this article, we propose an outlier detection approach in a multiple regression model using the properties of a difference-based variance estimator. This type of a difference-based variance estimator was originally used to estimate error variance in a non parametric regression model without estimating a non parametric function. This article first employed a difference-based error variance estimator to study the outlier detection problem in a multiple regression model. Our approach uses the leave-one-out type method based on difference-based error variance. The existing outlier detection approaches using the leave-one-out approach are highly affected by other outliers, while ours is not because our approach does not use the regression coefficient estimator. We compared our approach with several existing methods using a simulation study, suggesting the outperformance of our approach. The advantages of our approach are demonstrated using a real data application. Our approach can be extended to the non parametric regression model for outlier detection.

Non-parametric sequence-based learning approach for outlier detection in IoT

Although study on outlier detection techniques has long been an area of much research, few of those works relate to an Internet of Things (IoT) environment. In the last few years, with the advent of IoT and its numerous smart objects, data generated from sensors have increased exponentially. Since on the basis of these data many critical decisions are taken, it is therefore necessary to absolutely ensure its accuracy, correctness and integrity before any processing starts. Most algorithms in the past assumes outliers to be always an Error, that is caused by a malfunctioning sensor. However, this is not always true because an outlier could also be an important event that should not be neglected. This stresses the need to devise algorithms that is suited for an IoT environment which considers both Error, that is a result of faulty sensors or an Event, which is an indication of an abnormal phenomenon. This paper proposes a sequence based learning approach for outlier detection that works for both Error and Event. Simulations are performed on few benchmark datasets, a medical dataset and a real world dataset obtained through an experimental test bed. The results reveal exceptionally high accuracies with up to 99.65% for Error detection and 98.53% for Event detection.

Outlier detection approaches for wireless sensor networks: A survey

Over the past few years, wireless sensor networks have gained significant attention. They have been distributed in the real world in order to collect valuable raw sensed data. Due to high density, WSNs (Wireless Sensors Networks) are exposed to faults and nasty attacks. Likewise, the sensors readings are inaccurate and unreliable, which make Wireless Sensor Networks vulnerable to outliers. Abnormal data, outliers or anomalies are usually considered to be those sensor readings that have deviated significantly from normal behavior. However, the challenge is to ensure data quality, secure monitoring and reliable detection of interesting and critical events. In this survey, we describe a comprehensive overview of existing outlier detection techniques specifically used for the wireless sensor networks. Moreover, we present a comparative table used as a guideline to select which technique is adequate for the application in terms of characteristics such as detection mode, architectural structure and correlation extraction.

Multiple outliers detection in sparse high-dimensional regression

ABSTRACTThe presence of outliers would inevitably lead to distorted analysis and inappropriate prediction, especially for multiple outliers in high-dimensional regression, where the high dimensionality of the data might amplify the chance of an observation or multiple observations being outlying. Noting that the detection of outliers is not only necessary but also important in high-dimensional regression analysis, we, in this paper, propose a feasible outlier detection approach in sparse high-dimensional linear regression model. Firstly, we search a clean subset by use of the sure independence screening method and the least trimmed square regression estimates. Then, we define a high-dimensional outlier detection measure and propose a multiple outliers detection approach through multiple testing procedures. In addition, to enhance efficiency, we refine the outlier detection rule after obtaining a relatively reliable non-outlier subset based on the initial detection approach. By comparison studies based on Monte Carlo simulation, it is shown that the proposed method performs well for detecting multiple outliers in sparse high-dimensional linear regression model. We further illustrate the application of the proposed method by empirical analysis of a real-life protein and gene expression data.

Robust reduced-rank regression

SummaryIn high-dimensional multivariate regression problems, enforcing low rank in the coefficient matrix offers effective dimension reduction, which greatly facilitates parameter estimation and model interpretation. However, commonly used reduced-rank methods are sensitive to data corruption, as the low-rank dependence structure between response variables and predictors is easily distorted by outliers. We propose a robust reduced-rank regression approach for joint modelling and outlier detection. The problem is formulated as a regularized multivariate regression with a sparse mean-shift parameterization, which generalizes and unifies some popular robust multivariate methods. An efficient thresholding-based iterative procedure is developed for optimization. We show that the algorithm is guaranteed to converge and that the coordinatewise minimum point produced is statistically accurate under regularity conditions. Our theoretical investigations focus on non-asymptotic robust analysis, demonstrating that joint rank reduction and outlier detection leads to improved prediction accuracy. In particular, we show that redescending -functions can essentially attain the minimax optimal error rate, and in some less challenging problems convex regularization guarantees the same low error rate. The performance of the proposed method is examined through simulation studies and real-data examples.

An efficient approach for outlier detection in big sensor data of health care

SummaryIn recent years, wireless sensor networks are pervasive and are generating tons of data every second. Performing outlier detection to detect faulty sensors from such a large amount of data becomes a challenging task. Most of the existing techniques for outlier detection in wireless sensor networks concentrate only on contents of the data source without considering correlation among different data attributes. Moreover, these methods are not scalable to big data. To address these 2 limitations, this paper proposes an outlier detection approach based on correlation and dynamic SMO (sequential minimal optimization) regression that is scalable to big data. Initially, correlation is used to find out strongly correlated attributes and then the point anomalous nodes are detected using dynamic SMO regression. For fast processing of big data, Hadoop MapReduce framework is used. The experimental analysis demonstrates that the proposed approach efficiently detects the point and contextual anomalies and reduces the number of false alarms. For experiments, real data of sensors used in body sensor networks are taken from Physionet database.

Multivariate outlier detection in medicare claims payments applying probabilistic programming methods

The rising elderly population continues to demand more cost-effective healthcare programs. In particular, Medicare is a vital program serving the needs of the elderly in the United States. The growing number of people enrolled in healthcare programs such as Medicare, along with the enormous volume of money in the healthcare industry, increases the appeal for, and risk of, fraudulent activities. Out of the many possible factors for the rising cost of healthcare, fraud is a major contributor, but its impacts can be lessened through the use of fraud detection methods. In this paper, we assess possible illegitimate activities by looking at the amounts paid to providers for services rendered to patients. We propose a novel method for fraud detection that focuses on discovering outliers in Medicare payment data using multiple predictors as model inputs. Our multivariate outlier detection approach is twofold: (1) create a Multivariate Adaptive Regression Splines model to produce studentized residuals and, (2) use the residuals as input into a general univariate outlier detection model, based on full Bayesian inference, using probabilistic programming. Using this approach, we are able to incorporate multiple variables to detect outliers with a model that provides probability distributions, with credible intervals, rather than just point values, as with most common outlier detection methods. Additionally, these credible intervals further enhance confidence that the detected outliers should in fact be considered outlying values, thus possibly fraudulent activities. Our results show that the successful detection of these possibly fraudulent activities can provide effective and meaningful results for further investigation within various medical specialties.

A Data Mining Approach for Multivariate Outlier Detection in Postprocessing of Multitemporal InSAR Results

Displacement maps from multitemporal InSAR (MTI) are usually noisy and fragmented. Thresholding on ensemble coherence is a common practice for identifying radar scatterers that are less affected by decorrelation noise. Thresholding on coherence might, however, cause loss of information over the areas undergoing more complex deformation scenarios. If the discrepancies in the areas of moderate coherence share similar behavior, it appears important to take into account their spatial correlation for correct inference. The information over low-coherent areas might then be used in a similar way the coherence is used in thematic mapping applications such as change detection. We propose an approach based on data mining and statistical procedures for mitigating the impact of outliers in MTI results. Our approach allows for minimization of outliers in final results while preserving spatial and statistical dependence among observations. Tests from monitoring slope failures and undermined areas performed in this work have shown that this is beneficial: 1) for better evaluation of low-coherent scatterers that are commonly discarded by the standard thresholding procedure, 2) for tackling outlying observations with extremes in any variable, 3) for improving spatial densities of standard persistent scatterers, 4) for the evaluation of areas undergoing more complex deformation scenarios, and 5) for the visualization purposes.

LOMA: A local outlier mining algorithm based on attribute relevance analysis

Abstract In this study, we propose a novel local outlier detection approach - called LOMA - to mining local outliers in high-dimensional data sets. To improve the efficiency of outlier detection, LOMA prunes irrelevance attributes and objects in the data set by analyzing attribute relevance with a sparse factor threshold. Such a pruning technique substantially reduce the size of data sets. The core of LOMA is searching sparse subspace, which implements the particle swarm optimization method in reduced data sets. In the process of searching sparse subspace, we introduce the sparse coefficient threshold to represent sparse degrees of data objects in a subspace, where the data objects are considered as local outliers. The attribute relevance analysis provides a guidance for experts and users to identify useless attributes for detecting outliers. In addition, our sparse-subspace-based outlier algorithm is a novel technique for local-outlier detection in a wide variety of applications. Experimental results driven by both synthetic and UCI data sets validate the effectiveness and accuracy of our LOMA. In particular, LOMA achieves high mining efficiency and accuracy when the sparse factor threshold is set to a small value.

Cycle bases to the rescue

Cycle bases of graph theory are introduced for the analysis of transition data deposited in line-by-line rovibronic spectroscopic databases. The principal advantage of using cycle bases is that outlier transitions –almost always present in spectroscopic databases built from experimental data originating from many different sources– can be detected and identified straightforwardly and automatically. The data available for six water isotopologues, H216O, H217O, H218O, HD16O, HD17O, and HD18O, in the HITRAN2012 and GEISA2015 databases are used to demonstrate the utility of cycle-basis-based outlier-detection approaches. The spectroscopic databases appear to be sufficiently complete so that the great majority of the entries of the minimum cycle basis have the minimum possible length of four. More than 2000 transition conflicts have been identified for the isotopologue H216Oin the HITRAN2012 database, the seven common conflict types are discussed. It is recommended to employ cycle bases, and especially a minimum cycle basis, for the analysis of transitions deposited in high-resolution spectroscopic databases.