Outlier Detection Techniques Research Articles

Ranking with Distance based Outlier Detection Techniques: A Survey

Outlier Detection is very much popular in Data Mining field and it is an active research area due to its various applications like fraud detection, network sensor, email spam, stock market analysis, and intrusion detection and also in data cleaning. Here we will study some outlier detection technique which are mainly based on distance-based outlier detection with ranking approach and give some idea about the new technique which we will implement in future.

A Unified Framework for Outlier Detection in Trace Data Analysis

Process trace data (PTD) is an important data type in semiconductor manufacturing and has a very large aggregate volume. While data mining and statistical analysis play a key role in the quality control of wafers, the existence of outliers adversely affects the applications benefiting from PTD analysis. Due to the complexities of PTD and the resultant outlier patterns, this paper proposes a unified outlier detection framework which takes advantages of data complexity reduction using entropy and abrupt change detection using cumulative sum (CUSUM) method. To meet the practical needs of PTD analysis, a two-step algorithm taking into account of the related domain knowledge is developed, and its effectiveness is validated by using real PTD sets and a production example. The experimental results show that the proposed method outperforms the Fast Greedy Algorithm (FGA) and the Grubb's test, two commonly used outlier detection techniques for univariate data.

Wind Power Assessment Based on a WRF Wind Simulation with Developed Power Curve Modeling Methods

The accurate assessment of wind power potential requires not only the detailed knowledge of the local wind resource but also an equivalent power curve with good effect for a local wind farm. Although the probability distribution functions (pdfs) of the wind speed are commonly used, their seemingly good performance for distribution may not always translate into an accurate assessment of power generation. This paper contributes to the development of wind power assessment based on the wind speed simulation of weather research and forecasting (WRF) and two improved power curve modeling methods. These approaches are improvements on the power curve modeling that is originally fitted by the single layer feed-forward neural network (SLFN) in this paper; in addition, a data quality check and outlier detection technique and the directional curve modeling method are adopted to effectively enhance the original model performance. The proposed two methods, named WRF-SLFN-OD and WRF-SLFN-WD, are able to avoid the interference from abnormal output and the directional effect of local wind speed during the power curve modeling process. The data examined are from three stations in northern China; the simulation indicates that the two developed methods have strong abilities to provide a more accurate assessment of the wind power potential compared with the original methods.

An Efficient Intrusion Alerts Miner for Forensics Readiness in High Speed Networks

Intrusion Detection System is considered as a core tool in the collection of forensically relevant evidentiary data in real or near real time from the network. The emergence of High Speed Network (HSN) and Service oriented architecture/Web Services (SOA/WS) putted the IDS in face of a typical big data management problem. The log files that IDS generates are very enormous making very fastidious and both compute and memory intensive the forensics readiness process. Furthermore the high level rate of wrong alerts complicates the forensics expert alert analysis and it disproves its performance, efficiency and ability to select the best relevant evidences to attribute attacks to criminals. In this context, we propose Alert Miner (AM), an intrusion alert classifier, which classifies efficiently in near real-time the intrusion alerts in HSN for Web services. AM uses an outlier detection technique based on an adaptive deduced association rules set to classify the alerts automatically and without human assistance. AM reduces false positive alerts without losing high sensitivity (up to 95%) and accuracy up to (97%). Therefore AM facilitates the alert analysis process and allows the investigators to focus their analysis on the most critical alerts on near real-time scale and to postpone less critical alerts for an off-line log analysis.

Making sense of deviance: Identifying dissociating cases within the case series approach

The case series approach in cognitive neuropsychology provides a means to test theories that make quantitative predictions about associations between different components of the cognitive system [Schwartz, M. F., & Dell, G. S. (2010). Case series investigations in cognitive neuropsychology. Cognitive Neuropsychology, 27, 477–494]. However, even when the predicted association is borne out the study may include outliers—observations that deviate significantly from the rest of the data. These outliers may reveal individual cases whose cognitive impairments dissociate from other cases included in the study. These dissociating cases can pose a significant challenge to the theory being tested. Using a recent case series that investigated the underlying causes of letter perseveration in spelling [Fischer-Baum, S., & Rapp, B. (2012). Underlying cause(s) of letter perseveration errors. Neuropsychologia, 50, 305–318], I discuss statistical and theoretical issues that arise when using outlier detection techniques to identify dissociating cases in a case series study.

Automatic Growth Detection of Cell Cultures through Outlier Techniques using 2D Images

Using conventional statistics, we have developed a new method for cell culture analysis through outlier detection techniques. Statistical methods enable researchers in microbiology to identify experimental parameters that are critical for colony growth and inhibition. This paper reports a method for analysing 2D images of cell cultures in Petri dishs, such as fungi, bacteria or yeast. The aim of this study was to obtain a sensitive and robust method for detection of growth rate, surface coverage and the approximate number of cells in the colony. For testing we have implemented a software application called MoldATRIX. This software generates useful statistics and displays critical information about the cell colony area. Our results were obtained by analyzing a series of digital images of Aspergillus niger cultures at different time intervals. Moreover, our results show the behavior of Aspergillus niger on leather.Â

An online outlier detection method based on wavelet technique and robust RBF network

We focus on the issue of outlier detection for time-series data in a process control system (PCS), since outlier detection is a critical step before performing data-based system analysis. Several published articles have proved that a wavelet transform (WT) technique can be used to detect outliers in time-series data, but the standard WT detection method, as well as any other univariate outlier detection technique, does not distinguish between the sudden change caused by the changes of inputs and the fluctuations caused by outliers in PCS. In order to improve this shortcoming of the conventional WT method for the data in a PCS, a new algorithm combining the wavelet technique with a robust radial basis function (RBF) network is proposed here. In this method, a robust RBF network (RBFN) training algorithm is proposed, which can train the RBFN online using the original data as a training set without the need of clean data and thus fits the application of online detection. Furthermore, a hidden Markov model is adopted as an analysis tool to accomplish online automatic detection without pre-selecting the threshold. We compare the performance of our proposed method with the conventional wavelet method and the AR model method to demonstrate its validity through simulation and experimental applications to the data pretreatment process in an electric arc furnace electrode regulator system.

Outlier detection in reputation management system for P2P networks using rough set theory

ABSTRACTPeer‐to‐peer (P2P) networks are distributed, decentralized, dynamic networks that are self‐organized and self‐managed. P2P networks have emerged over the past several years as an effective and scalable medium for sharing distributed resources. However, determining the reliability and trustworthiness of the participating peers still remains a major security challenge. Reputation‐based trust management calculates peer trust as a measure of recommendations received from other peers. Malicious peers may give wrong reputation scores and also collude with other peers to make themselves or others appear trustworthy. In this paper, we propose the use of outlier detection technique to detect false testimony as outliers. We have applied rough set theory, an efficient and intelligent mathematical tool, to detect the outliers in the trust scores. We present the detailed methodology for implementing rough set theory for P2P network and detecting outlier scores in reputation metrics given by other peers and compared the model with the mechanism to detect outliers with the Eigen Trust model and eBay system. Trust computation without the outlier scores would be more accurate and enable proper verification and evaluation of peer trustworthiness. Copyright © 2013 John Wiley & Sons, Ltd.

Advancements of Outlier Detection: A Survey

Outlier detection is an important research problem in data mining that aims to discover useful abnormal and irregular patterns hidden in large datasets. In this paper, we present a survey of outlier detection techniques to reflect the recent advancements in this field. The survey will not only cov

Classification for Imbalanced and Overlapping Classes Using Outlier Detection and Sampling Techniques

In many real world applications, the example data among different pattern classes are imbalanced and overlapping, which hinder the classification performance of many learning algorithms. In this paper, data cleaning techniques based BNF (the borderline noise factor) is proposed to remove the borderline noise and three under-sampling methods are studied to select the representative majority class examples and remove the distant samples which are useless to form the decision boundary. BNF shows the degree of being a borderline noise and the outlier detection algorithm is improved to clean the whole dataset. Here G-mean (Geometric Mean) is used to define the threshold, which can improve the classification accuracy of minority classes while achieving better performance on the overall classification. The experimental results demonstrate the effectiveness of sampling method with data cleaning techniques based on BNF.

Outlier Detection Based On Similar Flocking Model In Wireless Sensor Networks

Abstract Outlier detection plays a crucial role in secure monitoring in Wireless Sensor Networks (WSN). Moreover, outlier detection techniques in WSN face the problem of limited resources of transmission bandwidth, energy consumption and storage capacity. In this paper, similar flocking model is proposed and a cluster algorithm based on similar flocking model (CASFM) is put forward to detect outliers in real-time stream data collected by sensor nodes. The similar flocking model improves the Vicsek model by introducing the similarity between individuals and velocity updating rule, which causes similar objects to cluster quickly. In order to save energy, CASFM algorithm preprocesses similar data on the sending sensors first, which greatly reduces the transmission of similar data. So the communication overhead is decreased. With the characteristics of self-organization and fast convergence of flocking model, stream data can be clustered quickly. The experimental results show that the proposed algorithm can detect outliers effectively with less energy consumption.

RODHA: Robust Outlier Detection using Hybrid Approach

The task of outlier detection is to find the small groups of data objects that are exceptional to the inherent behavior of the rest of the data. Detection of such outliers is fundamental to a variety of database and analytic tasks such as fraud detection and customer migration. There are several approaches[10] of outlier detection employed in many study areas amongst which distance based and density based outlier detection techniques have gathered most attention of researchers. In information theory, entropy is a core concept that measures uncertainty about a stochastic event, and it means that entropy describes the distribution of an event. Because of its ability to describe the distribution of data, entropy has been applied in clustering applications in data mining. In this paper, we have developed a robust supervised outlier detection algorithm using hybrid approach (RODHA) which incorporates both the concept of distance and density along with entropy measure while determining an outlier. We have provided an empirical study of different existing outlier detection algorithms and established the effectiveness of the proposed RODHA in comparison to other outlier detection algorithms.

Distributed online outlier detection in wireless sensor networks using ellipsoidal support vector machine

Low quality sensor data limits WSN capabilities for providing reliable real-time situation-awareness. Outlier detection is a solution to ensure the quality of sensor data. An effective and efficient outlier detection technique for WSNs not only identifies outliers in a distributed and online manner with high detection accuracy and low false alarm, but also satisfies WSN constraints in terms of communication, computational and memory complexity. In this paper, we take into account the correlation between sensor data attributes and propose two distributed and online outlier detection techniques based on a hyperellipsoidal one-class support vector machine (SVM). We also take advantage of the theory of spatio-temporal correlation to identify outliers and update the ellipsoidal SVM-based model representing the changed normal behavior of sensor data for further outlier identification. Simulation results show that our adaptive ellipsoidal SVM-based outlier detection technique achieves better detection accuracy and lower false alarm as compared to existing SVM-based techniques designed for WSNs.

Characteristics and classification of outlier detection techniques for wireless sensor networks in harsh environments: a survey

Wireless sensor networks (WSNs) have received considerable attention for multiple types of applications. In particular, outlier detection in WSNs has been an area of vast interest. Outlier detection becomes even more important for the applications involving harsh environments, however, it has not received extensive treatment in the literature. The identification of outliers in WSNs can be used for filtration of false data, find faulty nodes and discover events of interest. This paper presents a survey of the essential characteristics for the analysis of outlier detection techniques in harsh environments. These characteristics include, input data type, spatio-temporal and attribute correlations, user specified thresholds, outlier types(local and global), type of approach(distributed/centralized), outlier identification(event or error), outlier degree, outlier score, susceptibility to dynamic topology, non-stationarity and inhomogeneity. Moreover, the prioritization of various characteristics has been discussed for outlier detection techniques in harsh environments. The paper also gives a brief overview of the classification strategies for outlier detection techniques in WSNs and discusses the feasibility of various types of techniques for WSNs deployed in harsh environments.

FTDA: outlier detection‐based fault‐tolerant data aggregation for wireless sensor networks

ABSTRACTData aggregation protocols are essential for wireless sensor networks to prolong network lifetime by reducing energy consumption of sensor nodes. For mission‐critical wireless sensor networks, however, not only the energy consumption of sensor nodes but also the correctness of the data aggregation results is critical. As wireless sensor networks are usually deployed in harsh and hostile environments, malfunctioning and/or compromised sensor nodes negatively affect the correctness of the data aggregation results. This paper presents a fault‐tolerant data aggregation scheme that eliminates the false data sent by malfunctioning and/or compromised sensor nodes. To conserve energy while eliminating false data, an in‐network outlier detection technique that is based on locality sensitive hashing scheme is used. The simulation results show that the proposed scheme is able to reduce the number of false data transmissions, thereby increasing the data aggregation accuracy. Moreover, it is also observed that the proposed scheme reduces the overall data transmission in the network. Copyright © 2012 John Wiley & Sons, Ltd.

Multivariate whole genome average interval mapping: QTL analysis for multiple traits and/or environments

A major aim in some plant-based studies is the determination of quantitative trait loci (QTL) for multiple traits or across multiple environments. Understanding these QTL by trait or QTL by environment interactions can be of great value to the plant breeder. A whole genome approach for the analysis of QTL is presented for such multivariate applications. The approach is an extension of whole genome average interval mapping in which all intervals on a linkage map are included in the analysis simultaneously. A random effects working model is proposed for the multivariate (trait or environment) QTL effects for each interval, with a variance-covariance matrix linking the variates in a particular interval. The significance of the variance-covariance matrix for the QTL effects is tested and if significant, an outlier detection technique is used to select a putative QTL. This QTL by variate interaction is transferred to the fixed effects. The process is repeated until the variance-covariance matrix for QTL random effects is not significant; at this point all putative QTL have been selected. Unlinked markers can also be included in the analysis. A simulation study was conducted to examine the performance of the approach and demonstrated the multivariate approach results in increased power for detecting QTL in comparison to univariate methods. The approach is illustrated for data arising from experiments involving two doubled haploid populations. The first involves analysis of two wheat traits, α-amylase activity and height, while the second is concerned with a multi-environment trial for extensibility of flour dough. The method provides an approach for multi-trait and multi-environment QTL analysis in the presence of non-genetic sources of variation.

Outlier-Detection-Based Indoor Localization System for Wireless Sensor Networks

The paper exploits the outlier detection techniques for wireless-sensor-network- (WSN-) based localization problem and proposes an outlier detection scheme to cope with noisy sensor data. The cheap and widely available measurement technique—received signal strength (RSS)—is usually taken into account in the indoor localization system, but the RSS measurements are known to be sensitive to the change of the environment. The paper develops an outlier detection scheme to deal with abnormal RSS data so as to obtain more reliable measurements for localization. The effectiveness of the proposed approach is verified experimentally in an indoor environment.

A Decision Support System for Industry based upon Multivariate Spatial Outlier Detection Techniques

Decision support systems are computer based programs which assists decision makers in effective and efficient decision making. Outliers are the data sets which are highly irrelevant with respect to main data sets. Spatial outliers are the spatial objects with distinct features from their surrounding neighbours. In this paper we have proposed a decision support system for strategic decision makers to establish an industry. We have shown how spatial outlier detection techniques may be used to aid decision makers. Each site is evaluated based upon multiple decision variables. Here, we use two different outlier detection algorithms to detect a site which is most unsuitable to establish an industry. We also perform an experiment on a synthetic dataset of fifteen different sites to locate ambiguous sites. We found that decision making process for site selection may be improved by using spatial outlier detection techniques.

A System for Outlier Detection of High Dimensional Data

In high dimensional data large no of outliers are embedded in low dimensional subspaces known as projected outliers, but most of existing outlier detection techniques are unable to find these projected outliers, because these methods perform detection of abnormal patterns in full data space. So, outlier detection in high dimensional data becomes an important research problem. In this paper we are proposing an approach for outlier detection of high dimensional data. Here we are modifying the existing SPOT approach by adding three new concepts namely Adaption of Sparse Sub-Space Template (SST), Different combination of PCS parameters and set of non outlying cells for testing data set.

Reply to Rousselet et al.: A robust relationship between fiber density and memory recall

Using Pearson's correlation, we reported that fiber density between activated ventrolateral prefrontal cortex (VLPFC) and perirhinal cortex (PRC) and entorhinal cortex (ERC) correlated with memory recalled after deep and shallow encoding (1). Rousselet and Pernet (2) correctly point out that Pearson's correlation is susceptible to outliers. To circumvent this problem we used a random split half and showed that our main correlations were significant in both resulting subcohorts. Other statistical techniques for outlier detection confirm this robustness. Cook's distance estimates the influence that a single data point has on the result of a correlation or regression (3). With the conservative cutoff criterion of 4/N (4), Cook's distance showed no outliers in the deep study condition and one to three outliers in the shallow condition (which also showed weaker correlations in our original article). Without these outliers, all correlations remained significant (VLPFC–ERC shallow: r = 0.4255, P = 0.026; all other r > 0.627; all other P 0.532; all P < 0.0035).