Multivariate Outlier Detection Research Articles

Pointwise data depth for univariate and multivariate functional outlier detection

Pointwise data depth for univariate and multivariate functional outlier detection

Check your outliers! An introduction to identifying statistical outliers in R with easystats.

Beyond the challenge of keeping up to date with current best practices regarding the diagnosis and treatment of outliers, an additional difficulty arises concerning the mathematical implementation of the recommended methods. Here, we provide an overview of current recommendations and best practices and demonstrate how they can easily and conveniently be implemented in the R statistical computing software, using the {performance} package of the easystats ecosystem. We cover univariate, multivariate, and model-based statistical outlier detection methods, their recommended threshold, standard output, and plotting methods. We conclude by reviewing the different theoretical types of outliers, whether to exclude or winsorize them, and the importance of transparency. A preprint of this paper is available at: 10.31234/osf.io/bu6nt.

General value functions for fault detection in multivariate time series data.

One of the greatest challenges to the automated production of goods is equipment malfunction. Ideally, machines should be able to automatically predict and detect operational faults in order to minimize downtime and plan for timely maintenance. While traditional condition-based maintenance (CBM) involves costly sensor additions and engineering, machine learning approaches offer the potential to learn from already existing sensors. Implementations of data-driven CBM typically use supervised and semi-supervised learning to classify faults. In addition to a large collection of operation data, records of faulty operation are also necessary, which are often costly to obtain. Instead of classifying faults, we use an approach to detect abnormal behaviour within the machine's operation. This approach is analogous to semi-supervised anomaly detection in machine learning (ML), with important distinctions in experimental design and evaluation specific to the problem of industrial fault detection. We present a novel method of machine fault detection using temporal-difference learning and General Value Functions (GVFs). Using GVFs, we form a predictive model of sensor data to detect faulty behaviour. As sensor data from machines is not i.i.d. but closer to Markovian sampling, temporal-difference learning methods should be well suited for this data. We compare our GVF outlier detection (GVFOD) algorithm to a broad selection of multivariate and temporal outlier detection methods, using datasets collected from a tabletop robot emulating the movement of an industrial actuator. We find that not only does GVFOD achieve the same recall score as other multivariate OD algorithms, it attains significantly higher precision. Furthermore, GVFOD has intuitive hyperparameters which can be selected based upon expert knowledge of the application. Together, these findings allow for a more reliable detection of abnormal machine behaviour to allow ideal timing of maintenance; saving resources, time and cost.

Spatially Smoothed Robust Covariance Estimation for Local Outlier Detection

Most multivariate outlier detection procedures ignore the spatial dependency of observations, which is present in many real datasets from various application areas. This article introduces a new outlier detection method that accounts for a (continuously) varying covariance structure, depending on the spatial neighborhood of the observations. The underlying estimator thus constitutes a compromise between a unified global covariance estimation, and local covariances estimated for individual neighborhoods. Theoretical properties of the estimator are presented, in particular related to robustness properties, and an efficient algorithm for its computation is introduced. The performance of the method is evaluated and compared based on simulated data and for a dataset recorded from Austrian weather stations. Supplemental materials to the article are available online.

Dynamic Multivariate Outlier Detection Algorithm Using Ultraviolet Visible Spectroscopy for Monitoring Surface Water Contamination With Hydrological Fluctuation in Real-Time.

The contamination of surface water is of great harm. Ultraviolet-visible (UV-Vis) spectroscopy is an effective method to detect water contamination. However, surface water quality is influenced by hydrological fluctuation caused by rain, change of flow, etc., leading to changes of spectral characteristics over time. In the process of contamination detection, such changes cause confusion between hydrological fluctuation spectra and contaminated water spectra, thus increasing the false alarm rate. Besides, missing alarms of contaminated water is a common problem when the signal-to-noise ratio is low. In this paper, a dynamic multivariable outlier sampling rate detection (DM-SRD) algorithm is proposed. A dynamic updating strategy is introduced to increase adaptability to hydrological fluctuation. Additionally, multiple outlier variables are adopted as outlying degree indicators, which increases the accuracy of contamination detection. Two experiments were carried out using spectra collected from real surface water sites and hydrological fluctuation was constructed. To verify the effectiveness of the DM-SRD method, a comparison with the static SRD method and spectral match method was conducted. The results show that the accuracy of the DM-SRD method is 97.8%. Compared with the other two detection methods, DM-SRD significantly reduces false alarm rate and avoids missing alarms. Additionally, the results demonstrate that whether the database contained prior information on hydrological fluctuation or not, DM-SRD maintained high detection accuracy, which indicates great adaptability and robustness.

STAR_outliers: a python package that separates univariate outliers from non-normal distributions

There are not currently any univariate outlier detection algorithms that transform and model arbitrarily shaped distributions to remove univariate outliers. Some algorithms model skew, even fewer model kurtosis, and none of them model bimodality and monotonicity. To overcome these challenges, we have implemented an algorithm for Skew and Tail-heaviness Adjusted Removal of Outliers (STAR_outliers) that robustly removes univariate outliers from distributions with many different shape profiles, including extreme skew, extreme kurtosis, bimodality, and monotonicity. We show that STAR_outliers removes simulated outliers with greater recall and precision than several general algorithms, and it also models the outlier bounds of real data distributions with greater accuracy.Background Reliably removing univariate outliers from arbitrarily shaped distributions is a difficult task. Incorrectly assuming unimodality or overestimating tail heaviness fails to remove outliers, while underestimating tail heaviness incorrectly removes regular data from the tails. Skew often produces one heavy tail and one light tail, and we show that several sophisticated outlier removal algorithms often fail to remove outliers from the light tail. Multivariate outlier detection algorithms have recently become popular, but having tested PyOD’s multivariate outlier removal algorithms, we found them to be inadequate for univariate outlier removal. They usually do not allow for univariate input, and they do not fit their distributions of outliership scores with a model on which an outlier threshold can be accurately established. Thus, there is a need for a flexible outlier removal algorithm that can model arbitrarily shaped univariate distributions.Results In order to effectively model arbitrarily shaped univariate distributions, we have combined several well-established algorithms into a new algorithm called STAR_outliers. STAR_outliers removes more simulated true outliers and fewer non-outliers than several other univariate algorithms. These include several normality-assuming outlier removal methods, PyOD’s isolation forest (IF) outlier removal algorithm (ACM Transactions on Knowledge Discovery from Data (TKDD) 6:3, 2012) with default settings, and an IQR based algorithm by Verardi and Vermandele that removes outliers while accounting for skew and kurtosis (Verardi and Vermandele, Journal de la Société Française de Statistique 157:90–114, 2016). Since the IF algorithm’s default model poorly fit the outliership scores, we also compared the isolation forest algorithm with a model that entails removing as many datapoints as STAR_outliers does in order of decreasing outliership scores. We also compared these algorithms on the publicly available 2018 National Health and Nutrition Examination Survey (NHANES) data by setting the outlier threshold to keep values falling within the main 99.3 percent of the fitted model’s domain. We show that our STAR_outliers algorithm removes significantly closer to 0.7 percent of values from these features than other outlier removal methods on average.Conclusions STAR_outliers is an easily implemented python package for removing outliers that outperforms multiple commonly used methods of univariate outlier removal.

Mouse phenome database: curated data repository with interactive multi-population and multi-trait analyses.

The Mouse Phenome Database continues to serve as a curated repository and analysis suite for measured attributes of members of diverse mouse populations. The repository includes annotation to community standard ontologies and guidelines, a database of allelic states for 657 mouse strains, a collection of protocols, and analysis tools for flexible, interactive, user directed analyses that increasingly integrates data across traits and populations. The database has grown from its initial focus on a standard set of inbred strains to include heterogeneous mouse populations such as the Diversity Outbred and mapping crosses and well as Collaborative Cross, Hybrid Mouse Diversity Panel, and recombinant inbred strains. Most recently the system has expanded to include data from the International Mouse Phenotyping Consortium. Collectively these data are accessible by API and provided with an interactive tool suite that enables users' persistent selection, storage, and operation on collections of measures. The tool suite allows basic analyses, advanced functions with dynamic visualization including multi-population meta-analysis, multivariate outlier detection, trait pattern matching, correlation analyses and other functions. The data resources and analysis suite provide users a flexible environment in which to explore the basis of phenotypic variation in health and disease across the lifespan.

Multivariate functional outlier detection using the fast massive unsupervised outlier detection indices

We present definitions and properties of the fast massive unsupervised outlier detection (FastMUOD) indices, used for outlier detection (OD) in functional data. FastMUOD detects outliers by computing, for each curve, an amplitude, magnitude, and shape index meant to target the corresponding types of outliers. Some methods adapting FastMUOD to outlier detection in multivariate functional data are then proposed. These include applying FastMUOD on the components of the multivariate data and using random projections. Moreover, these techniques are tested on various simulated and real multivariate functional datasets. Compared with the state of the art in multivariate functional OD, the use of random projections showed the most effective results with similar, and in some cases improved, OD performance. Based on the proportion of random projections that flag each multivariate function as an outlier, we propose a new graphical tool, the magnitude‐shape‐amplitude (MSA) plot, useful for visualizing the magnitude, shape and amplitude outlyingness of multivariate functional data.

A further study comparing forward search multivariate outlier methods including ATLA with an application to clustering

This paper makes comparisons of automated procedures for robust multivariate outlier detection through discussion and simulation. In particular, automated procedures that use the forward search along with Mahalanobis distances to identify and classify multivariate outliers subject to predefined criteria are examined. Procedures utilizing a parametric model criterion based on a chi ^2-distribution are among these, whereas the multivariate Adaptive Trimmed Likelihood Algorithm (ATLA) identifies outliers based on an objective function that is derived from the asymptotics of the location estimator assuming a multivariate normal distribution. Several criterion including size (false positive rate), sensitivity, and relative efficiency are canvassed. To illustrate relative efficiency in a multivariate setting in a new way, measures of variability of the multivariate location parameter when the underlying distribution is chosen from a multivariate generalization of the Tukey–Huber epsilon -contamination model are used. Mean slippage models are also entertained. The simulation results here are illuminating and demonstrate there is no broadly accepted procedure that outperforms in all situations, albeit one may ascertain circumstances for which a particular method may be best if implemented. Finally the paper explores graphical monitoring for existence of clusters and the potential of classification through occurrence of multiple minima in the objective function using ATLA.

Multivariate Outlier Detection Based on Depth-Based Outlyingness Function

Multivariate Outlier Detection Based on Depth-Based Outlyingness Function

Multivariate Outlier Detection for Forest Fire Data Aggregation Accuracy

Wireless sensor networks have been a very important means in forest monitoring applications. A clustered sensor network comprises a set of cluster members and one cluster head. The cluster members are normally located close to each other, with overlaps among their sensing coverage within the cluster. The cluster members concurrently detect the same event to send to the Cluster Head node. This is where data aggregation is deployed to remove redundant data at the cost of data accuracy, where some data generated by the sensing process might be an outlier. Thus, it is important to conserve the aggregated data’s accuracy by performing an outlier data detection process before data aggregation is implemented. This paper concerns evaluating multivariate outlier detection (MOD) analysis on aggregated accuracy of data generated by a forest fire environment using OMNeT++ and performing the analysis in MATLAB R2018b. The findings of the study showed that the MOD algorithm conserved approximately 59.5% of aggregated data accuracy, compared with an equivalent algorithm, such as the FTDA algorithm, which conserved 54.25% of aggregated data accuracy for the same event.

Mahalanobis Distance Based Multivariate Outlier Detection to Improve Performance of Hypertension Prediction

Mahalanobis Distance Based Multivariate Outlier Detection to Improve Performance of Hypertension Prediction

A vibration-based approach for health monitoring of tie-rods under uncertain environmental conditions

This work proposes a vibration-based damage detection approach for axially loaded beam-type structures. The potential of the adopted approach is shown with an application to structural health monitoring of tie-rods: these elements, widely used in civil structures to balance horizontal forces in arches, are normally subject to a significant change of the axial load under environmental and operating conditions. The related changes in the modal parameters are usually greater than those caused by damage, at least at an early stage, making it hard to separate the different effects. In this work, the confounding effect of environmental factors is filtered out by considering more than one vibration mode at a time, thus framing damage detection as a multivariate outlier detection problem. Two damage indexes are presented, based on modal parameters (eigenfrequencies and mode shapes) and a multivariate metric (the Mahalanobis squared distance). The reasons behind the potential of the proposed framework are shown on simulated data first. Then, this strategy is validated on data coming from experimental tests, where two nominally identical tie-rods have been monitored for several months under the effects of realistic environmental and operational conditions. Both indexes proved to be weakly influenced by environmental variations, mainly related to temperature, and thus suitable for automatic damage detection.

Development of a Comprehensive Multi-Factor Method for Comparing Batting Performances in One-Day International Cricket

In cricket, one-day-international (ODI) batsmen have traditionally been compared on the dimensions of batting average (BA) and strike rate (SR). The conventional method of computing BA assumes that runs scored by a batsman follow an exponential or geometric distribution. This results in unreasonably equating batting inconsistency with batting mean. Our study shows that a Weibull distribution model gives a very sensible assessment of a batsman’s inconsistency, independent of his BA. It also provides a superior fit to batting scores of ODI batsmen. We also introduce a measure for ‘quality-runs’ scored by a batsman which takes into account the difficulty level of opposition. Additionally, longevity index and opposition diversity index are defined to make comparisons more holistic. A substantial amount of data engineering effort is made in segregating available data into home, away and neutral matches. The measures proposed in this paper are more comprehensive and granular than those found in the literature. Various combinations of these six criteria are used to rank a select group of great ODI batsmen by assigning objective weights derived from principal component analysis. Finally, multivariate statistical outlier detection procedure produces different lists of outstanding players corresponding to different combinations of criteria. Our proposed methodology may be gainfully used by a team management to select best batsmen in a given situation.

A New Procedure of Clustering Based on Multivariate Outlier Detection

Clustering is an extremely important task in a wide variety of ap plication domains especially in management and social science research. In this paper, an iterative procedure of clustering method based on multivariate outlier detection was proposed by using the famous Mahalanobis distance. At first, Mahalanobis distance should be calculated for the entire sample, then using T 2 -statistic fix a UCL. Above the UCL are treated as outliers which are grouped as outlier cluster and repeat the same procedure for the remaining inliers, until the variance-covariance matrix for the variables in the last cluster achieved singularity. At each iteration, multivariate test of mean used to check the discrimination between the outlier clusters and the inliers. Moreover, multivariate control charts also used to graphically visual izes the iterations and outlier clustering process. Finally multivariate test of means helps to firmly establish the cluster discrimination and validity. This paper employed this procedure for clustering 275 customers of a famous two wheeler in India based on 19 different attributes of the two wheeler and its company. The result of the proposed technique confirms there exist 5 and 7 outlier clusters of customers in the entire sample at 5% and 1% significance level respectively.

Assessing the statistical uniqueness of the Younger Dryas: a robust multivariate analysis

Abstract. During the last glacial period (ca. 120–11 kyr BP), dramatic temperature swings, known as Dansgaard–Oeschger (D–O) events, are clearly manifest in high-resolution oxygen isotope records from the Greenland Ice Sheet. Although variability in the Atlantic Meridional Overturning Circulation (AMOC) is often invoked, a unified explanation for what caused these “sawtooth-shaped” climate patterns has yet to be accepted. Of particular interest is the most recent D–O-shaped climate pattern that occurred from ∼ 14 600 to 11 500 years ago – the Bølling–Allerød (BA) warm interstadial and the subsequent Younger Dryas (YD) cold stadial. Unlike earlier D–O stadials, the YD is frequently considered a unique event, potentially resulting from a rerouting and/or flood of glacial meltwater into the North Atlantic or a meteorite impact. Yet, these mechanisms are less frequently considered as the cause of the earlier stadials. Using a robust multivariate outlier detection scheme – a novel approach for traditional paleoclimate research – we show that the pattern of climate change during the BA/YD is not statistically different from the other D–O events in the Greenland record and that it should not necessarily be considered unique when investigating the drivers of abrupt climate change. In so doing, our results present a novel statistical framework for paleoclimatic data analysis.

A New Procedure of Clustering Based on Multivariate Outlier Detection

A New Procedure of Clustering Based on Multivariate Outlier Detection

Monitoring Robust Estimates for Compositional Data

In a number of recent articles Riani, Cerioli, Atkinson and others advocate the technique of monitoring robust estimates computed over a range of key parameter values. Through this approach the diagnostic tools of choice can be tuned in such a way that highly robust estimators which are as efficient as possible are obtained. This approach is applicable to various robust multivariate estimates like S- and MM-estimates, MVE and MCD as well as to the Forward Search in whichmonitoring is part of the robust method. Key tool for detection of multivariate outliers and for monitoring of robust estimates is the Mahalanobis distances and statistics related to these distances. However, the results obtained with thistool in case of compositional data might be unrealistic since compositional data contain relative rather than absolute information and need to be transformed to the usual Euclidean geometry before the standard statistical tools can be applied. Various data transformations of compositional data have been introduced in the literature and theoretical results on the equivalence of the additive, the centered, and the isometric logratio transformation in the context of outlier identification exist. To illustrate the problem of monitoring compositional data and to demonstrate the usefulness of monitoring in this case we start with a simple example and then analyze a real life data set presenting the technologicalstructure of manufactured exports. The analysis is conducted with the R package fsdaR, which makes the analytical and graphical tools provided in the MATLAB FSDA library available for R users.

Estimating the Efforts of Mobile Application Development in the Planning Phase Using Nonlinear Regression Analysis

Abstract The authors consider the construction of a nonlinear multiple regression model, its confidence and prediction intervals to evaluate the efforts of mobile application development in the planning phase based on the multivariate normalizing transformation and outlier detection. The constructed model is compared to the linear regression model and nonlinear regression models based on the univariate transformations, such as the decimal logarithm, Box–Cox, and Johnson transformation. This model, in comparison with other regression models, has better prediction accuracy.

Outliers in official statistics

The purpose of this manuscript is to provide a survey on the important methods addressing outliers while producing official statistics. Outliers are often unavoidable in survey statistics. They may reduce the information of survey datasets and distort estimation on each step of the survey statistics production process. This paper defines outliers to be focused on each production step and introduces practical methods to cope with them. The statistical production process is roughly divided into the following three steps. The first step is data cleaning, and outliers to be focused are that may contain mistakes to be corrected. Robust estimators of a mean vector and covariance matrix are introduced for the purpose. The next step is imputation. Among a variety of imputation methods, regression and ratio imputation are the subjects in this paper. Outliers to be focused on in this step are not erroneous but have extreme values that may distort parameter estimation. Robust estimators that are not affected by remaining outliers are introduced. The final step is estimation and formatting. We have to be careful about outliers that have extreme values with large design weights since they have a considerable influence on the final statistics products. Weight calibration methods controlling the influence are discussed based on the robust weights obtained in the previous imputation step. A few examples of practical application are also provided briefly, although multivariate outlier detection methods introduced in this paper are mostly in the research stage in the field of official statistics.