Outlier Detection Techniques Research Articles

Application of Isolation Forest Technique for Outlier Detection in Water Quality Data

Application of Isolation Forest Technique for Outlier Detection in Water Quality Data

Research Issues of Outlier Detection in Trajectory Streams Using GPUs

The widespread availability of sensors like GPS and traffic cameras has made it possible to collect large amounts of spatio-temporal data. One such type of data are trajectories, each of which consists of a time-ordered sequence of positions that a moving object occupies in space as time goes by. Trajectories can be streamed in real time from sensors, and because of this, they capture the current state of moving objects. For this reason, trajectories can be used in applications such as the real-time detection of senior citizens who have just fallen or who have just gotten lost outdoors, the real-time detection of drunk drivers, and the real-time detection of enemy forces in the battlefield. These applications involve the identification of trajectories with anomalous behaviors, and require fast processing in order to take immediate preventive action. However, outlier detection poses challenges stemming from both the complexity of the data and of the task. One way to address this is through parallel architectures like GPUs. In this paper, we present the problem of outlier detection in trajectory streams, and discuss the research issues that should be addressed by new outlier detection techniques for trajectory streams on GPUs.

Introducing a new method for the fusion of fraud evidence in banking transactions with regards to uncertainty

Detection of fraudulent transactions is a vital factor for financial institutions, and finding more effective and accurate methods is of tremendous importance. The use of supervised data mining techniques is not feasible in many cases due to the lack of access to labeled data. Fraud detection is a complex task, and unsupervised methods like clustering and outlier detection techniques employed alone do not yield satisfactory results. Another issue is epistemic uncertainty due to the absence of sufficient information on the behavioral aspects of different customers, which also leads to poorer results for fraud detection and makes the fraud detection system inapplicable in real world environment. In this paper, using multi criteria decision method, intuitionistic fuzzy set, and evidential reasoning, a new method for detection of fraud was introduced, which infuses several behavioral evidence of a transaction concerning the effect of uncertainty for them. Transactional behavior was modeled by considering the trends of different main and aggregated variables at different periods and the extent of deviation of the new arrived transaction from each of these trends were considered as behavioral evidence. The final belief, which is the result of the combination of much evidence using the proposed method, will determine the originality of a newly arrived transaction. Finally, using a real world dataset, the results of the new method were compared with the results of Dempster–Shafer method in terms of the number of frauds discovered and the number of erroneous alerts they issued. The findings showed that the method introduced in this study has higher accuracy and lower false alarms compared to Dempster–Shafer method while the computational complexity of this method makes its implementation time longer.

Outlyingness: Which variables contribute most?

Outlier detection is an inevitable step to most statistical data analyses. However, the mere detection of an outlying case does not always answer all scientific questions associated with that data point. Outlier detection techniques, classical and robust alike, will typically flag the entire case as outlying, or attribute a specific case weight to the entire case. In practice, particularly in high dimensional data, the outlier will most likely not be outlying along all of its variables, but just along a subset of them. If so, the scientific question why the case has been flagged as an outlier becomes of interest. In this article, a fast and efficient method is proposed to detect variables that contribute most to an outlier's outlyingness. Thereby, it helps the analyst understand why an outlier lies out. The approach pursued in this work is to estimate the univariate direction of maximal outlyingness. It is shown that the problem of estimating that direction can be rewritten as the normed solution of a classical least squares regression problem. Identifying the subset of variables contributing most to outlyingness, can thus be achieved by estimating the associated least squares problem in a sparse manner. From a practical perspective, sparse partial least squares (SPLS) regression, preferably by the fast sparse NIPALS (SNIPLS) algorithm, is suggested to tackle that problem. The proposed methodology is illustrated to perform well both on simulated data and real life examples.

DIFFERENTIALLY PRIVATE OUTLIER DETECTION IN A COLLABORATIVE ENVIRONMENT.

Outlier detection is one of the most important data analytics tasks and is used in numerous applications and domains. The goal of outlier detection is to find abnormal entities that are significantly different from the remaining data. Often the underlying data is distributed across different organizations. If outlier detection is done locally, the results obtained are not as accurate as when outlier detection is done collaboratively over the combined data. However, the data cannot be easily integrated into a single database due to privacy and legal concerns. In this paper, we address precisely this problem. We first define privacy in the context of collaborative outlier detection. We then develop a novel method to find outliers from both horizontally partitioned and vertically partitioned categorical data in a privacy-preserving manner. Our method is based on a scalable outlier detection technique that uses attribute value frequencies. We provide an end-to-end privacy guarantee by using the differential privacy model and secure multiparty computation techniques. Experiments on real data show that our proposed technique is both effective and efficient.

Robust identification of target genes and outliers in triple-negative breast cancer data.

Correct classification of breast cancer subtypes is of high importance as it directly affects the therapeutic options. We focus on triple-negative breast cancer which has the worst prognosis among breast cancer types. Using cutting edge methods from the field of robust statistics, we analyze Breast Invasive Carcinoma transcriptomic data publicly available from The Cancer Genome Atlas data portal. Our analysis identifies statistical outliers that may correspond to misdiagnosed patients. Furthermore, it is illustrated that classical statistical methods may fail to identify outliers due to their heavy influence, prompting the need for robust statistics. Using robust sparse logistic regression we obtain 36 relevant genes, of which ca. 60% have been previously reported as biologically relevant to triple-negative breast cancer, reinforcing the validity of the method. The remaining 14 genes identified are new potential biomarkers for triple-negative breast cancer. Out of these, JAM3, SFT2D2, and PAPSS1 were previously associated to breast tumors or other types of cancer. The relevance of these genes is confirmed by the new DetectDeviatingCells outlier detection technique. A comparison of gene networks on the selected genes showed significant differences between triple-negative breast cancer and non-triple-negative breast cancer data. The individual role of FOXA1 in triple-negative breast cancer and non-triple-negative breast cancer, and the strong FOXA1-AGR2 connection in triple-negative breast cancer stand out. The goal of our paper is to contribute to the breast cancer/triple-negative breast cancer understanding and management. At the same time it demonstrates that robust regression and outlier detection constitute key strategies to cope with high-dimensional clinical data such as omics data.

Detection of Anomalies in Water Networks by Functional Data Analysis

A functional data analysis (FDA) based methodology for detecting anomalous flows in urban water networks is introduced. Primary hydraulic variables are recorded in real-time by telecontrol systems, so they are functional data (FD). In the first stage, the data are validated (false data are detected) and reconstructed, since there could be not only false data, but also missing and noisy data. FDA tools are used such as tolerance bands for FD and smoothing for dense and sparse FD. In the second stage, functional outlier detection tools are used in two phases. In Phase I, the data are cleared of anomalies to ensure that data are representative of the in-control system. The objective of Phase II is system monitoring. A new functional outlier detection method is also proposed based on archetypal analysis. The methodology is applied and illustrated with real data. A simulated study is also carried out to assess the performance of the outlier detection techniques, including our proposal. The results are very promising.

Secure Non-Consensus Based Spectrum Sensing in Non-Centralized Cognitive Radio Networks

In a non-centralized cognitive radio network, secondary users (SUs) cooperate in order to determine the presence of the primary user (PU) signal. This can be accomplished by each node sharing its spectrum sensing result with its neighboring SUs. In this paper, a non-consensus based distributed spectrum sensing (NCSS) scheme is presented, in which a node uses the raw energy level of the PU signal from its $h$ -hop neighbors for determining the status of the PU. A message passing scheme is used rather than the conventional consensus scheme. A comparison with an existing consensus algorithm is also shown. Further, two types of spectrum sensing data falsification attacks are considered in which the attackers inject very high or very low energy values. To catch such abnormal behavior of a node, an approach called secure NCSS algorithm is proposed, which isolates a node that generates extreme energy values so that it is removed from future computation. For this, an outlier detection technique is adopted. The approach is based on the idea that a malicious node’s energy level deviates significantly from those of genuine nodes, whereas all genuine nodes share nearby estimated energy levels.

Improving custom-tailored variability mining using outlier and cluster detection

• A variability mining approach to merge a set of models into a single representation. • The approach allows customization for different languages using the presented framework. • Supported by cluster and outlier detection technique to narrow input down to sensible subset. • Feasability case study with industrial case studies. To satisfy demand for customized software solutions, companies commonly use so-called clone-and-own approaches to reuse functionality by copying existing realization artifacts and modifying them to create new product variants. Lacking clear documentation about the variability relations (i.e., the common and varying parts), the resulting variants have to be developed, maintained and evolved in isolation. In previous work, we introduced a semi-automatic mining algorithm allowing custom-tailored identification of distinct variability relations for block-based model variants (e.g., MATLAB/Simulink models or statecharts) using user-adjustable metrics. However, variants completely unrelated with other variants (i.e., outliers) can negatively influence the usefulness of the generated variability relations for developers maintaining the variants (e.g., erroneous relations might be identified). In addition, splitting the compared models into smaller sets (i.e., clusters) can be sensible to provide developers separate view points on different variable system features. In further previous work, we proposed statistical clustering capable of identifying such outliers and clusters. The contribution of this paper is twofold. First, we present guidelines and a generic implementation that both ease adaptation of our variability mining algorithm for new languages. Second, we integrate our clustering approach as a preprocessing step to the mining. This allows users to remove outliers prior to executing variability mining on suggested clusters. Using models from two industrial case studies, we show feasibility of the approach and discuss how our clustering can support our variability mining in identifying sensible variability information.

Omnibus outlier detection in sensor networks using windowed locality sensitive hashing

Wireless Sensor Networks (WSNs) have become an integral part of cutting edge technological paradigms such as the Internet-of-Things (IoT) which incorporates a variety of smart application scenarios. WSNs include tiny sensors (motes), with constrained hardware capabilities and limited power supply that can collaboratively function in an unsupervised manner for a long period of time. Their purpose is to continuously monitor quantities of interest and provide answers to application queries. Sensor data streams are inherently spatiotemporal in nature, both because mote measurements form multidimensional time series and due to the spatial reference on the data based on the realm sensed by a mote. Motes are designed to be inexpensive, and thus sensory hardware is prone to temporary or permanent failures yielding faulty measurements. Such measurements may unpredictably forge a query answer, while truthful but abnormal mote samples may indicate undergoing phenomena. Therefore, outlier detection in sensor networks is of utmost importance.With limited power supply and communication being by far the main culprit in energy drain, outlier detection techniques in WSNs should achieve appropriate balance between reducing communication and providing real-time, continuously updated outlier reports. Prior works employ probabilistic or best effort approaches to accomplish the task, which either unpredictably compromise outlier detection accuracy or fail to explicitly tune the amount of communicated data. In this work, we introduce an omnibus outlier detection solution over spatiotemporally referenced sensor data that is capable of: (a) directly trading communication reduction for outlier detection quality with predictable accuracy guarantees, (b) accommodating both uni- and multi-dimensional outlier definitions, (c) operating under various streaming window models and (d) incorporating a wide variety of similarity measures to judge outliers.

DDC-Outlier: Preventing Medication Errors Using Unsupervised Learning.

Electronic health records have brought valuable improvements to hospital practices by integrating patient information. In fact, the understanding of these data can prevent mistakes that may put patients' lives at risk. Nonetheless, to the best of our knowledge, there are no previous studies addressing the automatic detection of outlier prescriptions, regarding dosage and frequency. In this paper, we propose an unsupervised method, called density-distance-centrality (DDC), to detect potential outlier prescriptions. A dataset with 563 thousand prescribed medications was used to assess our proposed approach against different state-of-the-art techniques for outlier detection. In the experiments, our approach achieves better results in the task of overdose and underdose detection in medical prescriptions, compared to other methods applied to this problem. Additionally, most of the false positive instances detected by our algorithm were potential prescriptions errors.

RES-Q: Robust Outlier Detection Algorithm for Fundamental Matrix Estimation

Detection of outliers present in noisy images for an accurate fundamental matrix estimation is an important research topic in the field of 3-D computer vision. Although a lot of research is conducted in this domain, not much study has been done in utilizing the robust statistics for successful outlier detection algorithms. This paper proposes to utilize a reprojection residual error-based technique for outlier detection. Given a noisy stereo image pair obtained from a pair of stereo cameras and a set of initial point correspondences between them, reprojection residual error and 3-sigma principle together with robust statistic-based $Q_{n}$ estimator (RES-Q) is proposed to efficiently detect the outliers and estimate the fundamental matrix with superior accuracy. The proposed RES-Q algorithm demonstrates greater precision and lower reprojection residual error than the state-of-the-art techniques. Moreover, in contrast to the assumption of Gaussian noise or symmetric noise model adopted by most previous approaches, the RES-Q is found to be robust for both symmetric and asymmetric random noise assumptions. The proposed algorithm is experimentally tested on both synthetic and real image data sets, and the experiments show that RES-Q is more effective and efficient than the classical outlier detection algorithms.

A Novel Attribute Based Dynamic Clustering with Schedule Based Rotation Method (ADC-SBR) for Outlier Detection

Detection of outliers in bank transactions has gained popularity in the recent years. The existing outlier detection techniques are unable to process the high volume of data. Hence, to address this issue, an efficient attribute-based dynamic clustering-schedule-based rotation (ADC-SBR) method is proposed. The similarity between transactions within a cluster is estimated using Jaccard coefficient-based labelling approach and the optimal cluster head is chosen by the similarity-based cluster head selection (SbCHS) method. The outlier detection is performed in two levels. The node level outlier detection is performed using linear regression model and the cluster level outlier detection is performed by deviation-based ranking. An own dataset with bank transactions is used for the experimental analysis. The suggested method is implemented in Apache Spark and is compared with existing algorithms for the metrics. The comparison results prove that the proposed method is optimal for all metrics than existing algorithms.

A Comparison of Outlier Detection Techniques for High-Dimensional Data

Outlier detection is a hot topic in machine learning. With the newly emerging technologies and diverse applications, the interest of outlier detection is increasing greatly. Recently, a significant number of outlier detection methods have been witnessed and successfully applied in a wide range of fields, including medical health, credit card fraud and intrusion detection. They can be used for conventional data analysis. However, it is not a trivial work to identify rare behaviors or patterns out from complicated data. In this paper, we provide a brief overview of the outlier detection methods for high-dimensional data, and offer comprehensive understanding of the-state-of-the-art techniques of outlier detection for practitioners. Specifically, we firstly summarize the recent advances on outlier detection for high-dimensional data, and then make an extensive experimental comparison to the popular detection methods on public datasets. Finally, several challenging issues and future research directions are discussed.

시계열 이상치 탐지 기법을 활용한 경부선 주요도시 철도 승객수의 이상치 탐색 연구

2004년 4월 1일, 국내 최초의 고속철도(HSR)인 KTX (Korea Train eXpress)가 경부선에 도 입 되었다. KTX의 등장은 경부선을 이용하는 철도 승객들의 운송수단 선택 및 도시구간별 이 용객 수 변화를 가져왔다. KTX의 등장과 같은 개입사건(Intervention events)의 영향은 개입사 건 전후 변화를 단순 통계량으로 분석하거나 개입 ARIMA 모델을 통해 분석 되었다. 개입 ARIMA 모델은 개입사건의 발생 시점(t)과 개입사건의 영향 형태(type) 등의 가정이 필요하다는 한계가 있었으며, 본 연구에서는 기존 연구에서의 한계점을 보완할 수 있는 시계열 이상치 탐지(time series outlier detection)를 활용하였다. 일반적으로 개입사건의 발생시기는 잘 알려져 있지 않으므로 시계열 이상치 탐지를 통해 개입사건에의 발생 시기를 추정할 수 있다. 시계열 이상치 탐지기법을 활용하여 개입의 시점과 영향 형태에 관한 가 정 없이 개입사건에 대한 영향을 분석할 수 있으며, 발생된 이상치의 시점을 개입사건의 시점, 이상치의 영향을 개입 사건의 영향으로 가정하였다. 데이터는 KTDB (Korea Transport Database)로 부터 KTX가 도입되기 이전인 2003 년부터 2014년까지 12년 동안의 경부선(4개의 주요 도시구간 합산)을 포함한 주요 도시구간 4개의 월별데이터를 수집하여 활용하였다. 경부선 도시 구간별 이상치를 탐지 하고 그 영향을 분석한 결과, 동일한 개입사건 임에도 그 영향의 형태의 정도가 도시구간마다 다르게 나타나거나 영향이 나타나지 않았으며, 기존 연구에서 분석되지 않은 개입사건을 찾을 수 있었다.

Feature Grouping-Based Outlier Detection Upon Streaming Trajectories

Outlier detection acts as one of the most important analysis tasks for trajectory stream. In stream scenarios, such properties as unlimitedness, time-varying evolutionary, sparsity, and skewness distribution of trajectories pose new challenges to outlier detection technique. Trajectory outlier detection techniques mainly focus on finding trajectory that is dissimilar to the majority of the others, which is based on the hypothesis that they are probably generated by a different mechanism. Most distance-based methods tend to utilize a function (e.g., weighted linear sum) to measure the similarity of two arbitrary objects provided that representative features have been extracted in advance. However, this kind of method is not tailored to identify the outlier which is close to its neighbors according to some features, but behaves significantly different from its neighbors in terms of the other features. To address this issue, we propose a feature grouping-based mechanism that divides all the features into two groups, where the first group ( Similarity Feature ) is used to find close neighbors and the second group ( Difference Feature ) is used to find outliers within the similar neighborhood. According to the feature differences among local adjacent objects in one or more time intervals, we present two outlier definitions, including local anomaly trajectory fragment ( TF-outlier ) and evolutionary anomaly moving object ( MO-outlier ). We devise a basic solution and then an optimized algorithm to detect both types of outliers. Experimental results show that our proposal is both effective and efficient to detect outliers upon trajectory data streams.

Resistant Statistical Methodologies for Anomaly Detection in Gas Turbine Dynamic Time Series: Development and Field Validation

The reliability of gas turbine (GT) health state monitoring and forecasting depends on the quality of sensor measurements directly taken from the unit. Outlier detection techniques have acquired a major importance, as they are capable of removing anomalous measurements and improve data quality. To this purpose, statistical parametric methodologies are widely employed thanks to the limited knowledge of the specific unit required to perform the analysis. The backward and forward moving window (BFMW) k–σ methodology proved its effectiveness in a previous study performed by the authors, to also manage dynamic time series, i.e., during a transient. However, the estimators used by the k–σ methodology are usually characterized by low statistical robustness and resistance. This paper aims at evaluating the benefits of implementing robust statistical estimators for the BFMW framework. Three different approaches are considered in this paper. The first methodology, k-MAD, replaces mean and standard deviation (SD) of the k–σ methodology with median and mean absolute deviation (MAD), respectively. The second methodology, σ-MAD, is a novel hybrid scheme combining the k–σ and the k-MAD methodologies for the backward and the forward windows, respectively. Finally, the biweight methodology implements biweight mean and biweight SD as location and dispersion estimators. First, the parameters of these methodologies are tuned and the respective performance is compared by means of simulated data. Different scenarios are considered to evaluate statistical efficiency, robustness, and resistance. Subsequently, the performance of these methodologies is further investigated by injecting outliers in field datasets taken on selected Siemens GTs. Results prove that all the investigated methodologies are suitable for outlier identification. Advantages and drawbacks of each methodology allow the identification of different scenarios in which their application can be most effective.

Non-parametric sequence-based learning approach for outlier detection in IoT

Although study on outlier detection techniques has long been an area of much research, few of those works relate to an Internet of Things (IoT) environment. In the last few years, with the advent of IoT and its numerous smart objects, data generated from sensors have increased exponentially. Since on the basis of these data many critical decisions are taken, it is therefore necessary to absolutely ensure its accuracy, correctness and integrity before any processing starts. Most algorithms in the past assumes outliers to be always an Error, that is caused by a malfunctioning sensor. However, this is not always true because an outlier could also be an important event that should not be neglected. This stresses the need to devise algorithms that is suited for an IoT environment which considers both Error, that is a result of faulty sensors or an Event, which is an indication of an abnormal phenomenon. This paper proposes a sequence based learning approach for outlier detection that works for both Error and Event. Simulations are performed on few benchmark datasets, a medical dataset and a real world dataset obtained through an experimental test bed. The results reveal exceptionally high accuracies with up to 99.65% for Error detection and 98.53% for Event detection.

A parameter based growing ensemble of self-organizing maps for outlier detection in healthcare

Outlier detection is critical for many applications such as healthcare, health insurance, medical diagnosis, predictive analytics, pattern recognition, intrusion detection, anomaly or defect detection, video surveillance, credit card fraud detection and text mining. Outlier detection techniques could be statistics, distance- or model based. Techniques, which are based on a single method for outlier detection usually have weaknesses and strengths and are mostly unstable. Outlier detection ensembles harness the strengths of individual detectors and result in stable performance. This paper presents a new parameter based growing self-organizing maps ensemble (GSOME) for outlier detection in multivariate patterns. For outlier detection, the proposed GSOME transforms non-linear relationships between high dimensional patterns into a simple 1D geometric relationship. Whatever the pattern dimensionality is, it is mapped to a single point of a line. The dispersion of mapped points will be used to locate the outliers and measure the degree of outlyingness. Several experiments on both real and synthetic data sets show the promising performance of the proposed GSOME.

Outlier detection approaches for wireless sensor networks: A survey

Over the past few years, wireless sensor networks have gained significant attention. They have been distributed in the real world in order to collect valuable raw sensed data. Due to high density, WSNs (Wireless Sensors Networks) are exposed to faults and nasty attacks. Likewise, the sensors readings are inaccurate and unreliable, which make Wireless Sensor Networks vulnerable to outliers. Abnormal data, outliers or anomalies are usually considered to be those sensor readings that have deviated significantly from normal behavior. However, the challenge is to ensure data quality, secure monitoring and reliable detection of interesting and critical events. In this survey, we describe a comprehensive overview of existing outlier detection techniques specifically used for the wireless sensor networks. Moreover, we present a comparative table used as a guideline to select which technique is adequate for the application in terms of characteristics such as detection mode, architectural structure and correlation extraction.