Outlier Detection Techniques Research Articles

A Graph Theory-Based Approach to Detect False Data Injection Attacks in Power System AC State Estimation

Nowadays, an energy management system (EMS) located at power grid control centers plays a critical role in ensuring the reliable operation of a grid. The SE serves as the main interface of EMS with the power grid, and because of that other EMS functionalities rely on its provided data. Recent studies show that the state estimator is vulnerable to various types of cyber-attacks. Among different types of such attacks, the false data injection attacks (FDIAs) are proven to be one of the most stealth ones and attracted much attention. Various countermeasures have been proposed to detect such kind of attacks. However, these approaches are generally based on some assumptions, such as being limited to detecting a specific type of FDIAs or the use of dc formulation for modeling the attack. In this article, a new algorithm is proposed to detect the FDIAs on ac state estimation. This algorithm applies outlier detection techniques on the state estimation results and then detects the possible attack by a graph-theory-based approach. The performance of the proposed algorithm is evaluated by simulating two main types of FDIAs in the IEEE 14 bus and 118 bus test systems. The results show that the algorithm performs satisfactorily in detecting both kinds of FDIAs with high rates.

Outlier detection in indoor localization and Internet of Things (IoT) using machine learning

In Internet of things (IoT) millions of devices are intelligently connected for providing smart services. Especially in indoor localization environment, that is one of the most concerning topic of smart cities, internet of things and wireless sensor networks. Many technologies are being used for localization purpose in indoor environment and Wi-Fi using received signal strengths (RSSs) is one of them. Wi-Fi RSSs are sensitive to reflection, refraction, interference and channel noise that cause irregularity in signal strengths. The irregular and anomalous RSS values, used in a Wi-Fi indoor localization environment, cannot define the location of any unknown node correctly. Therefore, this research has developed an outlier detection technique named as iF_Ensemble for Wi-Fi indoor localization environment by analyzing RSSs using the combination of supervised, unsupervised and ensemble machine learning methods. In this research isolation forest (iForest) is used as an unsupervised learning method. Supervised learning method includes support vector machine (SVM), K-nearest neighbor (KNN) and random forest (RF) classifiers with stacking that is an ensemble learning method. For the evaluation purpose accuracy, precision, recall, F-score and ROC-AUC curve are used. The evaluation of used machine learning method provides high accuracy of 97.8 percent with proposed outlier detection methods and almost 2 percent improvement in the accuracy of localization process in indoor environment after eliminating outliers.

Balanced binary search tree multiclass decomposition method with possible non-outliers

Multiclass decomposition algorithms are the means by which binary classification algorithms like support vector machine are used for multiclass classification problems. The popular multiclass decomposition algorithms like one against one (OAO), one against all (OAA), etc., perform the decomposition in a naive manner. This paper presents a novel heuristic-based decomposition algorithm that takes the Hausdorff distance between two classes to decide the decomposition. During the decomposition, rules are made to ensure a balanced binary search tree structure. To model the uncertainty and class noise present in the data, an unsupervised outlier detection technique has been used so that only possible non-outliers take part in the decomposition process. The presented algorithm has been evaluated and compared against OAO and OAA methods across 6 datasets. While evaluating the decomposition algorithms, fuzzy support vector machine has been used to model the class noise during each binary classification. The comparison shows that presented method not only provides comparable performance, but also in all cases, can classify the test samples with fewer average number of support vectors, thus leading to faster test performance. The paper further observes that the proposed approach can provide statistically better performance when the decomposition structure is learned only using the possible non-outliers, as compared to the scenario where the decomposition structure is learned using all samples.

A minimum spanning tree based partitioning and merging technique for clustering heterogeneous data sets

Clustering being an unsupervised learning technique, has been used extensively for knowledge discovery due to its less dependency on domain knowledge. Many clustering techniques were proposed in the literature to recognize the cluster of different characteristics. Most of them become inadequate either due to their dependency on user-defined parameters or when they are applied on multi-scale datasets. Hybrid clustering techniques have been proposed to take the advantage of both Partitional and Hierarchical techniques by first partitioning the dataset into several dense sub-clusters and merging them into actual clusters. However, the universality of the partition and merging criteria are not sufficient to capture many characteristics of the clusters. Minimum spanning tree (MST) has been used extensively for clustering because it preserves the intrinsic nature of the dataset even after the sparsification of the graph. In this paper, we propose a parameter-free, minimum spanning tree based efficient hybrid clustering algorithm to cluster the multi-scale datasets. In the first phase, we construct a MST of the dataset to capture the neighborhood information of data points and employ box-plot, an outlier detection technique on MST edge weights for effectively selecting the inconsistent edges to partition the data points into several dense sub-clusters. In the second phase, we propose a novel merging criterion to find the genuine clusters by iteratively merging only the pairs of adjacent sub-clusters. The merging technique involves both dis-connectivity and intra-similarity using the topology of two adjacent pairs which helps to identify the arbitrary shape and varying density clusters. Experiment results on various synthetic and real world datasets demonstrate the superior performance of the proposed technique over other popular clustering algorithms.

Multivariate Outlier Detection in Applied Data Analysis: Global, Local, Compositional and Cellwise Outliers

Outliers are encountered in all practical situations of data analysis, regardless of the discipline of application. However, the term outlier is not uniformly defined across all these fields since the differentiation between regular and irregular behaviour is naturally embedded in the subject area under consideration. Generalized approaches for outlier identification have to be modified to allow the diligent search for potential outliers. Therefore, an overview of different techniques for multivariate outlier detection is presented within the scope of selected kinds of data frequently found in the field of geosciences. In particular, three common types of data in geological studies are explored: spatial, compositional and flat data. All of these formats motivate new outlier concepts, such as local outlyingness, where the spatial information of the data is used to define a neighbourhood structure. Another type are compositional data, which nicely illustrate the fact that some kinds of data require not only adaptations to standard outlier approaches, but also transformations of the data itself before conducting the outlier search. Finally, the very recently developed concept of cellwise outlyingness, typically used for high-dimensional data, allows one to identify atypical cells in a data matrix. In practice, the different data formats can be mixed, and it is demonstrated in various examples how to proceed in such situations.

PERFORMANCE VALIDATION OF PRIOR QUANTIZATION TECHNIQUES IN OUTLIERS CLASSIFICATION USING WDBC DATASET

Data mining is the process of analyzing enormous data and summarizing it into the useful knowledge discovery and the task of data mining approaches is growing quickly, particularly classification techniques very efficient, way to classifying the data, which is important in the decision-making process for medical practitioners. This study presents the quantization and validation (OQV) techniques for fast outlier detection in large size WDBC data sets. The distance metrics utilization makes the algorithm as the linear one for various objects and assures the sequential scanning. The inclusion of direct quantization technique and the cluster explicit discovery assures the simplicity and the economical. The comparative analysis of proposed OQV techniques with the triangular boundary-based classification and the Weighing-based Feature Selection and Monotonic Classification (WFSMC) regarding the accuracy, precision, recall and the number of attributes assures an effectiveness of OQV for large size datasets.

The effects of the 2013 floods on Germany’s freight traffic

This article analyzes the spatio-temporal effect of the 2013 floods on freight traffic in Germany by using automatic traffic counter data. The methodology uses a proven time-series outlier detection and identification technique to endogenously determine if a counter was affected during the flood and estimate the magnitude and duration of the change in the number of vehicles passing through it. This is the first paper able to quantify climate-related variations in traffic across all the counters of a national network. Results show variations on 10% of all counters and 23% of all main roads. Results allow us to trace the configuration of disrupted and detour routes, recovery times, and the total effect on the network. Our findings serve as an input to other studies on the impact of exogenous events on the transport system and contribute towards the formulation of public policies to improve road resilience.

A Systematic Literature Review on Outlier Detection in Wireless Sensor Networks

A wireless sensor network (WSN) is defined as a set of spatially distributed and interconnected sensor nodes. WSNs allow one to monitor and recognize environmental phenomena such as soil moisture, air pollution, and health data. Because of the very limited resources available in sensors, the collected data from WSNs are often characterized as unreliable or uncertain. However, applications using WSNs demand precise readings, and uncertainty in data reading can cause serious damage (e.g., health monitoring data). Therefore, an efficient local/distributed data processing algorithm is needed to ensure: (1) the extraction of precise and reliable values from noisy readings; (2) the detection of anomalies from data reported by sensors; and (3) the identification of outlier sensors in a WSN. Several works have been conducted to achieve these objectives using several techniques such as machine learning algorithms, mathematical modeling, and clustering. The purpose of this paper is to conduct a systematic literature review to report the available works on outlier and anomaly detection in WSNs. The paper highlights works conducted from January 2004 to October 2018. A total of 3520 papers are reviewed in the initial search process. Later, these papers are filtered by title, abstract, and contents, and a total of 117 papers are selected. These papers are examined to answer the defined research questions. The current paper presents an improved taxonomy of outlier detection techniques. This will help researchers and practitioners to find the most relevant and recent studies related to outlier detection in WSNs. Finally, the paper identifies existing gaps that future studies can fill.

Characterizing Land Surface Phenology and Exotic Annual Grasses in Dryland Ecosystems Using Landsat and Sentinel-2 Data in Harmony

Invasive annual grasses, such as cheatgrass (Bromus tectorum L.), have proliferated in dryland ecosystems of the western United States, promoting increased fire activity and reduced biodiversity that can be detrimental to socio-environmental systems. Monitoring exotic annual grass cover and dynamics over large areas requires the use of remote sensing that can support early detection and rapid response initiatives. However, few studies have leveraged remote sensing technologies and computing frameworks capable of providing rangeland managers with maps of exotic annual grass cover at relatively high spatiotemporal resolutions and near real-time latencies. Here, we developed a system for automated mapping of invasive annual grass (%) cover using in situ observations, harmonized Landsat and Sentinel-2 (HLS) data, maps of biophysical variables, and machine learning techniques. A robust and automated cloud, cloud shadow, water, and snow/ice masking procedure (mean overall accuracy >81%) was implemented using time-series outlier detection and data mining techniques prior to spatiotemporal interpolation of HLS data via regression tree models (r = 0.94; mean absolute error (MAE) = 0.02). Weekly, cloud-free normalized difference vegetation index (NDVI) image composites (2016–2018) were used to construct a suite of spectral and phenological metrics (e.g., start and end of season dates), consistent with information derived from Moderate Resolution Image Spectroradiometer (MODIS) data. These metrics were incorporated into a data mining framework that accurately (r = 0.83; MAE = 11) modeled and mapped exotic annual grass (%) cover throughout dryland ecosystems in the western United States at a native, 30-m spatial resolution. Our results show that inclusion of weekly HLS time-series data and derived indicators improves our ability to map exotic annual grass cover, as compared to distribution models that use MODIS products or monthly, seasonal, or annual HLS composites as primary inputs. This research fills a critical gap in our ability to effectively assess, manage, and monitor drylands by providing a framework that allows for an accurate and timely depiction of land surface phenology and exotic annual grass cover at spatial and temporal resolutions that are meaningful to local resource managers.

Outlier detection and quasi-periodicity optimization algorithm: Frequency domain based outlier detection (FOD)

Outlier detection is one of the main challenges in the pre-processing stage of data analyses. In this study, we suggest a new non-parametric outlier detection technique which is based on the frequency-domain and Fourier Transform definitions and call it as the frequency-domain based outlier detection (FOD). From simulation results under various distributions and real data applications, we observe that our proposal approach is capable of detecting quasi-periodic outliers in time series data more successfully compared with other commonly used methods like z-score, box-plot and also faster than some specialized methods Grubbs method and autonomous anomaly detection (AAD) method. Therefore, we consider that our proposal approach can be an alternative approach to find quasi-periodic outliers in time series data.

Outlier Detection and Effects on Modeling

In this work, a comprehensive framework for traditional outlier detection techniques based on simple and multiple linear regression models was studied. Two data sets were used for the illustration and evaluation of each class of outlier detection techniques (analytical and graphical methods). Outlier detection aims at identifying such outlier in order to improve the analytic of data and suitable model built. Furthermore, comparisons of the different methods were done to highlight the advantages, disadvantages and performance issues of each class of outlier detection techniques. The results show that by removing the influential points (or outliers), the model adequacy increased (from R2 = 0.72 to R2 = 0.97). It was observed that Jackknife residuals and Atkinson’s measure methods are very useful in detecting outliers; hence, both methods were recommended for outliers’ detection.

Road Traffic Outlier Detection Technique based on Linear Regression

Abstract Road traffic anomaly detection is a very important aspect of intelligent traffic management system. Traffic anomaly may arise due to several reasons like unusual traffic incidents and malfunctioning of sensors deployed over the road network to capture traffic information. Unusual traffic incident includes road accident, road blockage due to construction, any major events and so on. For smooth mobility of the citizens, it is very important to identify these kinds of scenarios with minimum delay so that traffic management authority can take proper measures. This paper proposes a technique based on statistical model which identifies the temporal outliers in the road traffic. Z-score and linear regression model are two statistical models have been used in combination for detection of temporal outliers. The proposed technique can be used to detect unusual traffic incident or sensors failure.

A novel attribute-based dynamic clustering with schedule-based rotation method for outlier detection

Detection of outliers in bank transactions has gained popularity in the recent years. The existing outlier detection techniques are unable to process the high volume of data. Hence, to address this issue, an efficient attribute-based dynamic clustering-schedule-based rotation (ADC-SBR) method is proposed. The similarity between transactions within a cluster is estimated using Jaccard coefficient-based labelling approach and the optimal cluster head is chosen by the similarity-based cluster head selection (SbCHS) method. The outlier detection is performed in two levels. The node level outlier detection is performed using linear regression model and the cluster level outlier detection is performed by deviation-based ranking. An own dataset with bank transactions is used for the experimental analysis. The suggested method is implemented in Apache Spark and is compared with existing algorithms for the metrics. The comparison results prove that the proposed method is optimal for all metrics than existing algorithms.

Behavioural Intrusion Detection in Water Distribution Systems Using Neural Networks

There has been an increasing number of attacks against critical water system infrastructure in recent years. This is largely due to the fact that these systems are heavily dependent on computer networks meaning that an attacker can use conventional techniques to penetrate this network which would give them access to the supervisory control and data acquisition (SCADA) system. The devastating impact of a successful attack in these critical infrastructure applications could be long-lasting with major social and financial implications. Intrusion detection systems are deployed as a secondary defence mechanism in case an attacker is able to bypass the systems preventative security mechanisms. In this thesis, behavioural intrusion detection is addressed in the context of detecting cyber-attacks in water distribution systems. A comparative analysis of various predictive neural network architectures is conducted and from this a novel voting-based ensemble technique is presented. Finally an analysis of how this approach to behavioural intrusion detection can be enhanced by both univariate and multivariate outlier detection techniques It was found that multiple algorithms working together are able to counteract their limitation to produce a more robust algorithm with improved results.

Identifying Extreme Rainfall Events Using Functional Outliers Detection Methods

Outlier detection techniques play a vital role in exploring unusual data of extreme events that have a critical effect considerably in the modeling and forecasting of functional data. The functional methods have an effective way of identifying outliers graphically, which might not be visible through the original data plot in classical analysis. This study’s main objective is to detect the extreme rainfall events using functional outliers detection methods depending on the depth and density functions. In order to identify the unusual events of rainfall variation over long time intervals, this work conducts based on the average monthly rainfall of the Taiz region from 1998 to 2019. Data were extracted from the Tropical Rainfall Measuring Mission and the analysis has been processed by R software. The approaches applied in this study involve rainbow plots, functional highest density region box-plot as well as functional bag-plot. According to the current results, the functional density box-plot method has proven effective in detecting outlier compared to the functional depth bag-plot method. In conclusion, the results of the current study showed that the rainfall over the Taiz region during the last two decades was influenced by the extreme events of years 1999, 2004, 2005, and 2009.

Online sentiment towards iconic species

Studies assessing online public sentiment towards biodiversity conservation are almost non-existent. The use of social media data and other online data sources is increasing in conservation science. We collected social media and online news data pertaining to rhinoceros, which are iconic species especially threatened by illegal wildlife trade, and assessed online sentiment towards these species using natural language processing methods. We also used an outlier detection technique to identify the most prominent conservation-related events imprinted into this data. We found that tragic events, such as the death of the last male northern white rhinoceros, Sudan, in March 2018, triggered the strongest reactions, which appeared to be concentrated in western countries, outside rhinoceros range states. We also found a strong temporal cross-correlation between social media data volume and online news volume in relation to tragic events, while other events only appeared in either social media or online news. Our results highlight that the public is concerned about biodiversity loss and this, in turn, can be used to increase pressure on decision makers to develop adequate conservation actions that can help reverse the biodiversity crisis. The proposed methods and analyses can be used to infer sentiment towards any biodiversity topic from digital media data, and to detect which events are perceived most important to the public.

A method for outlier detection based on cluster analysis and visual expert criteria

AbstractOutlier detection is an important problem occurring in a wide range of areas. Outliers are the outcome of fraudulent behaviour, mechanical faults, human error, or simply natural deviations. Many data mining applications perform outlier detection, often as a preliminary step in order to filter out outliers and build more representative models. In this paper, we propose an outlier detection method based on a clustering process. The aim behind the proposal outlined in this paper is to overcome the specificity of many existing outlier detection techniques that fail to take into account the inherent dispersion of domain objects. The outlier detection method is based on four criteria designed to represent how human beings (experts in each domain) visually identify outliers within a set of objects after analysing the clusters. This has an advantage over other clustering‐based outlier detection techniques that are founded on a purely numerical analysis of clusters. Our proposal has been evaluated, with satisfactory results, on data (particularly time series) from two different domains: stabilometry, a branch of medicine studying balance‐related functions in human beings and electroencephalography (EEG), a neurological exploration used to diagnose nervous system disorders. To validate the proposed method, we studied method outlier detection and efficiency in terms of runtime. The results of regression analyses confirm that our proposal is useful for detecting outlier data in different domains, with a false positive rate of less than 2% and a reliability greater than 99%.

Incremental rare pattern based approach for identifying outliers in medical data

Outlier detection is an imperative field of data mining that has several applications in the field of medical research. Mining outliers based on the notion of rare patterns can be a promising solution for medical diagnosis as it attempts to identify the unconventional and abnormal risk patterns present in medical data. A crucial issue in medical data analysis is the continuous growth of medical databases due to the addition of new records. Existing outlier detection techniques are capable of handling only static data and thus re-execute from scratch to identify the outliers from incremental medical data. This paper introduces an efficient rare pattern based outlier detection (RPOD) method that identifies outliers by mining rare patterns from incremental data. To avoid multiple database scans and expensive candidate generation steps performed by existent rare pattern mining techniques and facilitate incremental mining, a single pass prefix tree-based rare pattern mining technique is proposed. The proposed rare pattern mining technique is a modification of the well-known FP-Growth frequent pattern mining algorithm. Furthermore, to identify the outliers based on the set of generated rare patterns, an outlier detection technique is also presented. The significance of proposed RPOD approach is demonstrated using several well-known medical datasets. Comparative performance evaluation substantiates the predominance of RPOD approach over existing outlier mining methods.

Evaluation of robust outlier detection methods for zero-inflated complex data

ABSTRACTOutlier detection can be seen as a pre-processing step for locating data points in a data sample, which do not conform to the majority of observations. Various techniques and methods for outlier detection can be found in the literature dealing with different types of data. However, many data sets are inflated by true zeros and, in addition, some components/variables might be of compositional nature. Important examples of such data sets are the Structural Earnings Survey, the Structural Business Statistics, the European Statistics on Income and Living Conditions, tax data or – as in this contribution – household expenditure data which are used, for example, to estimate the Purchase Power Parity of a country.In this work, robust univariate and multivariate outlier detection methods are compared by a complex simulation study that considers various challenges included in data sets, namely structural (true) zeros, missing values, and compositional variables. These circumstances make it difficult or impossible to flag true outliers and influential observations by well-known outlier detection methods.Our aim is to assess the performance of outlier detection methods in terms of their effectiveness to identify outliers when applied to challenging data sets such as the household expenditures data surveyed all over the world. Moreover, different methods are evaluated through a close-to-reality simulation study. Differences in performance of univariate and multivariate robust techniques for outlier detection and their shortcomings are reported. We found that robust multivariate methods outperform robust univariate methods. The best performing methods in finding the outliers and in providing a low false discovery rate were found to be the generalized S estimators (GSE), the BACON-EEM algorithm and a compositional method (CoDa-Cov). In addition, these methods performed also best when the outliers are imputed based on the corresponding outlier detection method and indicators are estimated from the data sets.

Parallel graph-based anomaly detection technique for sequential data

In data mining, outlier detection is applied in different domains. It has very large applications such as energy consumption analysis, forecasting hurricanes in meteorological data, fraud and intrusion detection, event detection and system monitoring in sensor networks, etc. Most of existing outlier detection techniques depend on the properties of a particular type of data and can not deal with a large volume of data well, which mean that there is a necessity for improved methodologies and techniques to be applied to a large amount of data with different types in other application areas. In this paper, a parallel outlier detection technique is developed to detect the outliers in the sequential data. Although there are many types of outliers, this paper concentrates on the contextual anomalies. The proposed technique uses a graph approach to detect the outliers. It is very flexible, fast, and no labeled data is needed comparing to many previous approaches. The experimental results show the detected contextual outliers in the sequential data, as well as the efficient scaling up to handle the massive data by increasing the number of processors. The results prove that the parallelism of the proposed technique is very valuable.