Outlier Detection Approach Research Articles

A local density-based approach for outlier detection

This paper presents a simple and effective density-based outlier detection approach with local kernel density estimation (KDE). A Relative Density-based Outlier Score (RDOS) is introduced to measure local outlierness of objects, in which the density distribution at the location of an object is estimated with a local KDE method based on extended nearest neighbors of the object. Instead of using only k nearest neighbors, we further consider reverse nearest neighbors and shared nearest neighbors of an object for density distribution estimation. Some theoretical properties of the proposed RDOS including its expected value and false alarm probability are derived. A comprehensive experimental study on both synthetic and real-life data sets demonstrates that our approach is more effective than state-of-the-art outlier detection methods.

Image noise Detection and Removal based on Enhanced GridLOF Algorithm

Image noise removal is a major task in image processing where noise can harness any information inferred from the image especially when the noise level is high. Although there exists many outlier detection approaches used for this task, more enhancements are needed to achieve better performance specifically in terms of time. This paper proposes a new algorithm to detect and remove noise from images depending on an enhanced version of GridLOF algorithm. The enhancement aims to reduce the time and complexity of the algorithm while attaining comparable accuracy. Simulation results on a set of different images proved that proposed algorithm achieves the standard accuracy.

Robust Structure and Motion Recovery Based on Augmented Factorization

This paper proposes a new strategy to promote the robustness of structure from motion algorithm from uncalibrated video sequences. First, an augmented affine factorization algorithm is formulated to circumvent the difficulty in image registration with noise and outliers contaminated data. Then, an alternative weighted factorization scheme is designed to handle the missing data and measurement uncertainties in the tracking matrix. Finally, a robust strategy for structure and motion recovery is proposed to deal with outliers and large measurement noise. This paper makes the following main contributions: 1) An augmented factorization algorithm is proposed to circumvent the difficult image registration problem of previous affine factorization, and the approach is applicable to both rigid and nonrigid scenarios; 2) by employing the fact that image reprojection residuals are largely proportional to the error magnitude in the tracking data, a simple outliers detection approach is proposed; and 3) a robust factorization strategy is developed based on the distribution of the reprojection residuals. Furthermore, the proposed approach can be easily extended to nonrigid scenarios. Experiments using synthetic and real image data demonstrate the robustness and efficiency of the proposed approach over previous algorithms.

An Unsupervised Approach for Combining Scores of Outlier Detection Techniques, Based on Similarity Measures

Outlier detection, the discovery of observations that deviates from normal behavior, has become crucial in many application domains. Numerous and diverse algorithms have been proposed to detect them. These algorithms identify outliers using precise definitions of the concept of outliers, thus their performance depends largely on the context of application. The construction of ensembles has been proposed as a solution to increase the individual capacity of each algorithm. However, the unsupervised scenario (absence of class labels) in the domains where outlier detection operates restricts the use of approaches relying on the existence of labels. In this paper, two novel unsupervised approaches using ensembles of heterogeneous types of detectors are proposed. Both approaches construct the ensemble using solely the results produced by each algorithm, identifying and giving more weight to the most suitable techniques depending on the particular dataset under examination. Through experimental evaluation in real world datasets, we demonstrate that our proposed algorithm provides a significant improvement over the base algorithms and even over existing approaches for ensemble outlier detection.

Neighborhood relevant outlier detection approach based on information entropy

Neighborhood relevant outlier detection approach based on information entropy

EXPERIENCE

Enterprise's archives are inevitably affected by the presence of data quality problems (also called glitches). This article proposes the application of a new method to analyze the quality of datasets stored in the tables of a database, with no knowledge of the semantics of the data and without the need to define repositories of rules. The proposed method is based on proper revisions of different approaches for outlier detection that are combined to boost overall performance and accuracy. A novel transformation algorithm is conceived that treats the items in database tables as data points in real coordinate space of n dimensions, so that fields containing dates and fields containing text are processed to calculate distances between those data points. The implementation of an iterative approach ensures that global and local outliers are discovered even if they are subject, primarily in datasets with multiple outliers or clusters of outliers, to masking and swamping effects. The application of the method to a set of archives, some of which have been studied extensively in the literature, provides very promising experimental results and outperforms the application of a single other technique. Finally, a list of future research directions is highlighted.

A Survey On Outlier Detection Technique In Streaming Data Using Data Clustering Approach

Data mining is a highly researched area in the today’s world as data is crucial part of many application, due to which many researchers express their interest in this domain. As there arises a need to process large dataset which imposes different challenges for researchers. To have a data which is free from a noisy attributes , known as a filtered data , is of much important to gain accuracy in a result sets. For that , finding and eliminate the noisy objects has gained a much more importance. An object that does not follow the footprints of usual data object is called outliers. Outlier detection process is used in numerous applications like fraud detection, intrusion detection system, tracking environmental activities, healthcare diagnosis. Numbers of approaches are used in the process of detection of outlier. Most approaches focuses to use Cluster-based and Distance based approach (i.e. using KMeans algorithm and Euclidian distance) for outlier detection in data sets which help them to create a group of similar elements or cluster of data points. Clustering techniques are highly useful for grouping similar data items from data sets and after that by applying distance based calculations, detection of outlier is done, so they are called cluster-based outlier detection. KMeans and Euclidian distance are the most common and popular algorithm for clustering and outlier detection process due to its simplicity and efficiency. Different application areas of outlier detection are discussed in this paper.

Mobile-Assisted Anchor Outlier Detection for Localization in Wireless Sensor Networks

Accurate location information is critical to many applications in wireless sensor networks (WSNs) such as target tracking, environmental monitoring and geographical routing. Localization aims to figure out the locations of unknown nodes based on global locations of anchors and inter-node distance measurements. However, the existence of outlier anchors and outlier distances degrade localization accuracy in many localization algorithms. Most existing outlier detection approaches focus on distance outlier detection; few efforts have been devoted to anchor outlier detection. In this paper, we propose a mobile-assisted approach to detect outlier anchors and mitigate their negative effects in localization to achieve high localization accuracy. The proposed approach, namely Mobile-Assisted Anchor Outlier detection (MAAO),employs a mobile element to traverse the wireless sensor network several times to collect position information from static anchors in the network. For every static anchor, the mobile element computes the average of the anchor’s positions acquired from all tours, and compare it with the position acquired from the last mobile tour to detect whether the anchor is an outlier or not. The evaluation results show that MAAO can effectively detect outlier anchors, which consequently results in remarkable improvement in localization accuracy by not using outlier anchors in the localization process.

Bagged Subspaces for Unsupervised Outlier Detection

In many domains, important events are not represented as the common scenario, but as deviations from the rule. The importance and impact associated with these particular, outnumbered, deviant, and sometimes even previously unseen events is directly related to the application domain (e.g., breast cancer detection, satellite image classification, etc.). The detection of these rare events or outliers has recently been gaining popularity as evidenced by the wide variety of algorithms currently available. These algorithms are based on different assumptions about what constitutes an outlier, a characteristic pointing toward their integration in an ensemble to improve their individual detection rate. However, there are two factors that limit the use of current ensemble outlier detection approaches: first, in most cases, outliers are not detectable in full dimensionality, but instead are located in specific subspaces of data; and second, despite the expected improvement on detection rate achieved using an ensemble of detectors, the computational efficiency of the ensemble will increase linearly as the number of components increases. In this article, we propose an ensemble approach that identifies outliers based on different subsets of features and subsamples of data, providing more robust results while improving the computational efficiency of similar ensemble outlier detection approaches.

Detection of Outlier in Uncategorical Dataset using Hybrid Algorithm

Outlier mining is an important task of discovering the data records which have an exceptional behavior comparing with other records in the remaining dataset. Outliers do not follow with other data objects in the dataset. There are many effective approaches to detect outliers in numerical data. Most of the earliest work on outlier detection was performed by the statistics community on numeric data. But for categorical dataset there are limited approaches By using NAVF (Normally distributed attribute value frequency) and ROAD (Ranking- based Outlier Analysis and Detection algorithm) and new hybrid approach for outlier detection in categorical dataset will be formed.

Model-Based Outlier Detection System with Statistical Preprocessing

Reliability, lack of error, and security are important improvements to quality of service. Outlier detection is a process of detecting the erroneous parts or abnormal objects in defined populations, and can contribute to secured and error-free services. Outlier detection approaches can be categorized into four types: statistic-based, unsupervised, supervised, and semi-supervised. A model-based outlier detection system with statistical preprocessing is proposed, taking advantage of the statistical approach to preprocess training data and using unsupervised learning to construct the model. The robustness of the proposed system is evaluated using the performance evaluation metrics sum of squared error (SSE) and time to build model (TBM). The proposed system performs better for detecting outliers regardless of the application domain.

False Discovery Versus Familywise Error Rate Approaches to Outlier Detection

Outliers, in general, are observations that deviate sufficiently from a base distribution. This study deals with outlier detection approaches for large samples from continuous univariate distributions. Investigated are the properties of a practical newer outlier detection approach based on use of a false discovery rate (FDR) method in conjunction with a robustly estimated Tukey g-and-h base distribution. Compared are the properties of a boxplot type outlier detection approach that controls the familywise error rate (FWER) with a newer FDR approach. These options are compared in terms of error rates and effects of moving the outliers gradually further from base distribution center while using 5% and 1% FDR or FWERs. Two microarray datasets are used as examples where the assumed null distributions do not fit the data well. In such cases, the proposed estimated Tukey g-and-h null distribution approach leads to superior outlier detection performance. Supplementary materials for this article are available online.

Demographically-Based Evaluation of Genomic Regions under Selection in Domestic Dogs.

Controlling for background demographic effects is important for accurately identifying loci that have recently undergone positive selection. To date, the effects of demography have not yet been explicitly considered when identifying loci under selection during dog domestication. To investigate positive selection on the dog lineage early in the domestication, we examined patterns of polymorphism in six canid genomes that were previously used to infer a demographic model of dog domestication. Using an inferred demographic model, we computed false discovery rates (FDR) and identified 349 outlier regions consistent with positive selection at a low FDR. The signals in the top 100 regions were frequently centered on candidate genes related to brain function and behavior, including LHFPL3, CADM2, GRIK3, SH3GL2, MBP, PDE7B, NTAN1, and GLRA1. These regions contained significant enrichments in behavioral ontology categories. The 3rd top hit, CCRN4L, plays a major role in lipid metabolism, that is supported by additional metabolism related candidates revealed in our scan, including SCP2D1 and PDXC1. Comparing our method to an empirical outlier approach that does not directly account for demography, we found only modest overlaps between the two methods, with 60% of empirical outliers having no overlap with our demography-based outlier detection approach. Demography-aware approaches have lower-rates of false discovery. Our top candidates for selection, in addition to expanding the set of neurobehavioral candidate genes, include genes related to lipid metabolism, suggesting a dietary target of selection that was important during the period when proto-dogs hunted and fed alongside hunter-gatherers.

Survey on Outlier Detection in Data Stream

mining provides a way for finding hidden and useful knowledge from the large amount of data .usually we find any information by finding normal trends or distribution of data .But sometimes rare event or data object may provide information which is very interesting to us .Outlier detection is one of the task of data mining .It finds abnormal data point or sequence hidden in the dataset .Data stream is unbounded sequence of data with explicit or implicit temporal context .Data stream is uncertain and dynamic in nature. Traditional outlier detection techniques for static data which require whole dataset for modelling is not suitable for data stream because whole data stream cannot be stored. Network intrusion detection ,web click stream analysis ,fraud detection ,fault detection in machines ,sensor data analysis are some of the applications of data stream outlier detection .In this paper, we have described several issues in data stream outlier detection and usual approaches or techniques for finding outlier in data stream .

Distance outlier detection with a mobile beacon in wireless sensor networks localisation

Range-based localisation algorithms rely on distance measurements between nodes. If outlier distances exist, they make the estimated positions deviate from the true positions. Thus, it is important to detect and remove outlier distance measurements to achieve high localisation accuracy. In this paper, we propose a novel outlier detection approach called middle distance mobile beacon placement MDMP. Different from the traditional triangle-inequality-based approach that can detect only the existence of outlier distances but cannot distinguish which measurements are outliers, MDMP can exactly find which distances are outliers and exclude them from the localisation process. This prevents nodes from getting wrong position estimates caused by these outlier measurements. Simulation results show that MDMP improves the outlier detection accuracy by identifying more outliers than the traditional triangle inequality violation TIV-based method does. The evaluation results also show remarkable improvement in location accuracy by deleting outliers with MDMP.

Outlier Detection in Energy Disaggregation Using Subspace Learning and Gaussian Mixture Model

Special Complex non-Gaussian processes may have dynamic operation scenario shifts so that the traditional Outlier detection approaches become ill-suited. This paper proposes a new outlier detection approach based on using subspace learning and Gaussian mixture model(GMM) in energy disaggregation. Locality preserving projections(LPP) of subspace learning can optimally preserve the neighborhood structure, reveal the intrinsic manifold structure of the data and keep outliers far away from the normal sample compared with the principal component analysis (PCA). The results show proposed approach can significantly improve performance of outlier detection in energy disaggregation, increase the fraction true-positive from 93.8% to 97%, decrease the fraction false-positive from 35.48% to 25.8%.

A practical outlier detection approach for mixed-attribute data

Outlier detection in mixed-attribute space is a challenging problem for which only a few approaches have been proposed. However, such existing methods suffer from the fact that there is a lack of an automatic mechanism to formally discriminate between outliers and inliers. In fact, a common approach to outlier identification is to estimate an outlier score for each object and then provide a ranked list of points, expecting outliers to come first. A major problem of such an approach is where to stop reading the ranked list? How many points should be chosen as outliers? Other methods, instead of outlier ranking, implement various strategies that depend on user-specified thresholds to discriminate outliers from inliers. Ad-hoc threshold values are often used. With such an unprincipled approach it is impossible to be objective or consistent. To alleviate these problems, we propose a principled approach based on the bivariate beta mixture model to identify outliers in mixed-attribute data. The proposed approach is able to automatically discriminate outliers from inliers and it can be applied to both mixed-type attribute and single-type (numerical or categorical) attribute data without any feature transformation. Our experimental study demonstrates the suitability of the proposed approach in comparison to mainstream methods.

A decomposition of the outlier detection problem into a set of supervised learning problems

Outlier detection methods automatically identify instances that deviate from the majority of the data. In this paper, we propose a novel approach for unsupervised outlier detection, which re-formulates the outlier detection problem in numerical data as a set of supervised regression learning problems. For each attribute, we learn a predictive model which predicts the values of that attribute from the values of all other attributes, and compute the deviations between the predictions and the actual values. From those deviations, we derive both a weight for each attribute, and a final outlier score using those weights. The weights help separating the relevant attributes from the irrelevant ones, and thus make the approach well suitable for discovering outliers otherwise masked in high-dimensional data. An empirical evaluation shows that our approach outperforms existing algorithms, and is particularly robust in datasets with many irrelevant attributes. Furthermore, we show that if a symbolic machine learning method is used to solve the individual learning problems, the approach is also capable of generating concise explanations for the detected outliers.

Fuzzy Improved Decision Tree Approach for Outlier Detection in SMS

Spam is one of the serious problems faced by internet community globally. Spam Detection is a critical issue in business world. In this paper an intelligent three stage model is presented to perform the spam inclusive outlier identification. The SMS textual dataset is taken as input and than its filtration is done. After that this textual information is converted to the statistical information using fuzzy and assign the weights to dataset. The decision tree algorithm is than applied on this fuzzy weighed dataset to classify the dataset. This algorithm is defined to separate the spam and non spam data values. A comparison of existing Bayesian and proposed Fuzzy based decision tree approach is done. The results shows that the recognition rate is improved using the proposed approach. The work is implemented in weka integrated java environment.

A Modified Approach for Detection of Outliers

Tukey’s boxplot is very popular tool for detection of outliers. It reveals the location, spread and skewness of the data. It works nicely for detection of outliers when the data are symmetric. When the data are skewed it covers boundary away from the whisker on the compressed side while declares erroneous outliers on the extended side of the distribution. Hubert and Vandervieren (2008) made adjustment in Tukey’s technique to overcome this problem. However another problem arises that is the adjusted boxplot constructs the interval of critical values which even exceeds from the extremes of the data. In this situation adjusted boxplot is unable to detect outliers. This paper gives solution of this problem and proposed approach detects outliers properly. The validity of the technique has been checked by constructing fences around the true 95% values of different distributions. Simulation technique has been applied by drawing different sample size from chi square, beta and lognormal distributions. Fences constructed by the modified technique are close to the true 95% than adjusted boxplot which proves its superiority on the existing technique.