Existing Outlier Detection Techniques Research Articles

ReVISOR: ResUNets with visibility and intensity for structured outlier removal

Laser acquisition of large-scale point clouds is prone to several categories of measurement errors, which can lead, in particular, to the presence of undesirable outlier points. Existing outlier detection techniques are primarily based on analyzing local properties of point distributions to distinguish “clean” from “noisy” data. In contrast, real-world acquisition often has to deal with reflective surfaces, which can give rise to structured outliers that can be indistinguishable from clean geometry through purely local analysis. We make several contributions to address the problem of reflection-induced outlier detection. First, to overcome the scarcity of annotated data, we introduce a new dataset tailored for this task. Second, to capture non-local dependencies, we study and demonstrate, for the first time, the utility of deep learning based semantic segmentation architectures for reflection-induced outlier detection. By doing so, we bring together the fields of shape denoising/repair and semantic segmentation. Third, we demonstrate that additional non-local cues in the form of laser intensity and a computed visibility signal help boost the performance considerably. We denote our pipeline as ResUNets with Visibility and Intensity for Structured Outlier Removal, or ReVISOR, and demonstrate its superior performance against existing baselines on real-world data.

A novel attribute-based dynamic clustering with schedule-based rotation method for outlier detection

Detection of outliers in bank transactions has gained popularity in the recent years. The existing outlier detection techniques are unable to process the high volume of data. Hence, to address this issue, an efficient attribute-based dynamic clustering-schedule-based rotation (ADC-SBR) method is proposed. The similarity between transactions within a cluster is estimated using Jaccard coefficient-based labelling approach and the optimal cluster head is chosen by the similarity-based cluster head selection (SbCHS) method. The outlier detection is performed in two levels. The node level outlier detection is performed using linear regression model and the cluster level outlier detection is performed by deviation-based ranking. An own dataset with bank transactions is used for the experimental analysis. The suggested method is implemented in Apache Spark and is compared with existing algorithms for the metrics. The comparison results prove that the proposed method is optimal for all metrics than existing algorithms.

A method for outlier detection based on cluster analysis and visual expert criteria

AbstractOutlier detection is an important problem occurring in a wide range of areas. Outliers are the outcome of fraudulent behaviour, mechanical faults, human error, or simply natural deviations. Many data mining applications perform outlier detection, often as a preliminary step in order to filter out outliers and build more representative models. In this paper, we propose an outlier detection method based on a clustering process. The aim behind the proposal outlined in this paper is to overcome the specificity of many existing outlier detection techniques that fail to take into account the inherent dispersion of domain objects. The outlier detection method is based on four criteria designed to represent how human beings (experts in each domain) visually identify outliers within a set of objects after analysing the clusters. This has an advantage over other clustering‐based outlier detection techniques that are founded on a purely numerical analysis of clusters. Our proposal has been evaluated, with satisfactory results, on data (particularly time series) from two different domains: stabilometry, a branch of medicine studying balance‐related functions in human beings and electroencephalography (EEG), a neurological exploration used to diagnose nervous system disorders. To validate the proposed method, we studied method outlier detection and efficiency in terms of runtime. The results of regression analyses confirm that our proposal is useful for detecting outlier data in different domains, with a false positive rate of less than 2% and a reliability greater than 99%.

Incremental rare pattern based approach for identifying outliers in medical data

Outlier detection is an imperative field of data mining that has several applications in the field of medical research. Mining outliers based on the notion of rare patterns can be a promising solution for medical diagnosis as it attempts to identify the unconventional and abnormal risk patterns present in medical data. A crucial issue in medical data analysis is the continuous growth of medical databases due to the addition of new records. Existing outlier detection techniques are capable of handling only static data and thus re-execute from scratch to identify the outliers from incremental medical data. This paper introduces an efficient rare pattern based outlier detection (RPOD) method that identifies outliers by mining rare patterns from incremental data. To avoid multiple database scans and expensive candidate generation steps performed by existent rare pattern mining techniques and facilitate incremental mining, a single pass prefix tree-based rare pattern mining technique is proposed. The proposed rare pattern mining technique is a modification of the well-known FP-Growth frequent pattern mining algorithm. Furthermore, to identify the outliers based on the set of generated rare patterns, an outlier detection technique is also presented. The significance of proposed RPOD approach is demonstrated using several well-known medical datasets. Comparative performance evaluation substantiates the predominance of RPOD approach over existing outlier mining methods.

Parallel graph-based anomaly detection technique for sequential data

In data mining, outlier detection is applied in different domains. It has very large applications such as energy consumption analysis, forecasting hurricanes in meteorological data, fraud and intrusion detection, event detection and system monitoring in sensor networks, etc. Most of existing outlier detection techniques depend on the properties of a particular type of data and can not deal with a large volume of data well, which mean that there is a necessity for improved methodologies and techniques to be applied to a large amount of data with different types in other application areas. In this paper, a parallel outlier detection technique is developed to detect the outliers in the sequential data. Although there are many types of outliers, this paper concentrates on the contextual anomalies. The proposed technique uses a graph approach to detect the outliers. It is very flexible, fast, and no labeled data is needed comparing to many previous approaches. The experimental results show the detected contextual outliers in the sequential data, as well as the efficient scaling up to handle the massive data by increasing the number of processors. The results prove that the parallelism of the proposed technique is very valuable.

A scalable correlation‐based approach for outlier detection in wireless body sensor networks

SummaryOutlier detection is one of the prominent research domain in the field of data mining and big data analytics. Nowadays, most of the data in healthcare centers are remotely monitored and are generated from different wireless sensors. The core objective of outlier detection in this domain is the recognition of the true physiologically anomalous data and the anomalies because of faulty sensors. In real healthcare monitoring scenario, various sensors are related to each other. So, while detecting outliers in wireless body sensor networks (WBSNs), correlation among different sensor nodes is of major concern. Most of the existing outlier detection techniques consider the sensors to be linearly correlated, which may not always be the case in real life applications. The traditional techniques for outlier detection are also not scalable to big data. To address the above issues, in this paper, we propose an approach for outlier detection that is scalable to big data and also handles the nonlinearly correlated attributes efficiently. The proposed approach is implemented on Hadoop map reduce framework for the rapid processing of big data. The evaluation results are validated using the simulated dataset of WBSNs taken from the Physionet library. The results are compared with various existing outlier detection approaches and demonstrated that the proposed approach is more effective in spotting the physiological outliers and sensor anomalies accurately.

A Novel Attribute Based Dynamic Clustering with Schedule Based Rotation Method (ADC-SBR) for Outlier Detection

Detection of outliers in bank transactions has gained popularity in the recent years. The existing outlier detection techniques are unable to process the high volume of data. Hence, to address this issue, an efficient attribute-based dynamic clustering-schedule-based rotation (ADC-SBR) method is proposed. The similarity between transactions within a cluster is estimated using Jaccard coefficient-based labelling approach and the optimal cluster head is chosen by the similarity-based cluster head selection (SbCHS) method. The outlier detection is performed in two levels. The node level outlier detection is performed using linear regression model and the cluster level outlier detection is performed by deviation-based ranking. An own dataset with bank transactions is used for the experimental analysis. The suggested method is implemented in Apache Spark and is compared with existing algorithms for the metrics. The comparison results prove that the proposed method is optimal for all metrics than existing algorithms.

Outlier detection approaches for wireless sensor networks: A survey

Over the past few years, wireless sensor networks have gained significant attention. They have been distributed in the real world in order to collect valuable raw sensed data. Due to high density, WSNs (Wireless Sensors Networks) are exposed to faults and nasty attacks. Likewise, the sensors readings are inaccurate and unreliable, which make Wireless Sensor Networks vulnerable to outliers. Abnormal data, outliers or anomalies are usually considered to be those sensor readings that have deviated significantly from normal behavior. However, the challenge is to ensure data quality, secure monitoring and reliable detection of interesting and critical events. In this survey, we describe a comprehensive overview of existing outlier detection techniques specifically used for the wireless sensor networks. Moreover, we present a comparative table used as a guideline to select which technique is adequate for the application in terms of characteristics such as detection mode, architectural structure and correlation extraction.

Outlier Detection Techniques for Localization in Wireless Sensor Networks: A Survey

In wireless sensor networks (WSNs), localization is one of the most important topics because the location information is typically useful for many applications. The primary data used in a localization process include the locations of anchor nodes and the distances between neighboring nodes. However, these data may contain outliers that deviate from their true values. The existence of the outliers might make the estimated positions not accurate. Thus, it is important to detect and handle outliers in order to achieve high localization accuracy. In this paper, we survey the existing outlier detection techniques for localization in wireless sensor networks. We provide taxonomy for classifying outlier detection techniques in WSNs localization based on different features. In addition, we present comparisons of these techniques. Finally, we discuss the future research directions in this area.

A Survey on Outlier Detection in Financial Transactions

Outlier detection is a very important concept in the data mining. It is useful in data analysis. Nowadays, a direct mapping can be found between the data outliers and real world anomalies. Hence the outlier detection techniques can be applied to detect the abnormal activities in the real world. Outlier detection has been researched within various application domains and knowledge disciplines. This survey provides an overview of existing outlier detection techniques that can be applied in the financial domain. It mainly focuses on the idea of detecting the suspicious or outlier financial transactions.

A System for Outlier Detection of High Dimensional Data

In high dimensional data large no of outliers are embedded in low dimensional subspaces known as projected outliers, but most of existing outlier detection techniques are unable to find these projected outliers, because these methods perform detection of abnormal patterns in full data space. So, outlier detection in high dimensional data becomes an important research problem. In this paper we are proposing an approach for outlier detection of high dimensional data. Here we are modifying the existing SPOT approach by adding three new concepts namely Adaption of Sparse Sub-Space Template (SST), Different combination of PCS parameters and set of non outlying cells for testing data set.

Outlier Detection Techniques for Wireless Sensor Networks: A Survey

In the field of wireless sensor networks, those measurements that significantly deviate from the normal pattern of sensed data are considered as outliers. The potential sources of outliers include noise and errors, events, and malicious attacks on the network. Traditional outlier detection techniques are not directly applicable to wireless sensor networks due to the nature of sensor data and specific requirements and limitations of the wireless sensor networks. This survey provides a comprehensive overview of existing outlier detection techniques specifically developed for the wireless sensor networks. Additionally, it presents a technique-based taxonomy and a comparative table to be used as a guideline to select a technique suitable for the application at hand based on characteristics such as data type, outlier type, outlier identity, and outlier degree.