Outlier Labeling Research Articles

An optimized outlier detection function for multibeam echo-sounder data

Multibeam sonar data are a valuable tool for seafloor mapping and geological studies. However, the presence of outliers in multibeam data can distort the results of analyses and reduce the accuracy of seafloor maps. In this paper, we define a weighting function based on the performance of various outlier detection techniques (ODTs) for detecting outliers in multibeam data, which calculates an outlier probability score for each sounding. Our results show that each ODT has its own strengths and weaknesses, and that a combination of outlier detection techniques is promising to improve reproducibility, explainability and the accuracy of the detection process. To address the challenge of detecting outliers in multibeam data, we propose a weighted outlier detection function that outperforms individual outlier detection techniques in terms of precision, recall and F1 scores by considering their strengths and combining them in a way that accounts for variations in the data. The function detects various types of outliers with high precision and recall values, resulting in valuable improvements in outlier detection performance for multibeam data. Overall, our proposed workflow has the potential to significantly improve the way multibeam data cleaning is performed, with the weighted outlier detection function being applied first, detecting most of the outlier automatically, followed by a domain-expert review of a small group of soundings whose automatic outlier labeling is not unequivocal.

Penetration rate prediction in heterogeneous formations: A geomechanical approach through machine learning

Bourgoyne and Young Method (BYM) is one of the most widely used rate of penetration (ROP) prediction methods. Drilling data, in this method, must be taken from uniform-lithology sections (homogeneous formations) to get the maximum prediction accuracy using multiple linear regression analysis. In the presence of heterogeneity, the accuracy of BYM decreases substantially due to complexity in lithology. There are various studies on ROP prediction based on BYM. Since these studies considers uniform lithology only, none of them functions satisfactorily in heterogeneous formations. Implementing a different type of regression model as an alternative to multiple linear regression by modifying BYM to predict ROP in heterogeneous environment for well programming is the main purpose of this study. This is done by introducing several geomechanical parameters to the BYM to estimate a relationship between the rock mechanical properties and the heterogeneity. An offset drilling data is taken from a heterogeneous formation and converted into BYM's functions. The effect of feature selection, data size, and different outlier labeling approaches on ROP prediction accuracy in heterogeneous formations are investigated by applying several predictive techniques: multiple linear regression, support vector regression, and artificial neural networks. The prediction results are compared and examined from a statistical point of view. It is revealed that the BYM could not represent ROP effectively in a heterogeneous environment. As demonstrated for the first time, the ROP prediction accuracy in heterogeneous formations significantly increases when machine learning techniques are used together with additional features comprising heterogeneity.

A conceptual framework to deal with outliers in ecology

Research on ecology commonly involves the need to face datasets that contain extreme or unusual observations. The presence of outliers during data analysis has been of concern for researchers generating a lot of discussion on different methods and strategies on how to deal with them and became a recurrent issue of interest in debate forums. Systematic elimination or data transformation could lead to ignore important ecological processes and draw wrong conclusions. The importance of coping with extreme observations during data analysis in ecology becomes clear in the context of relevant environmental aspects such as impact assessment, pest control, and biodiversity conservation. In those contexts, misinterpretation of results due to an incorrect processing of outliers may difficult decision making or even lead to failing to adopt the best management program. In this work, I summarized different approaches to deal with extreme observations such as outlier labeling, accommodation, and identification, using calculation and visualization methods, and provide a conceptual workflow as a general overview for data analysis.

An Improved Boxplot for Univariate Data

ABSTRACTThe boxplot is an effective data-visualization tool useful in diverse applications and disciplines. Although more sophisticated graphical methods exist, the boxplot remains relevant due to its simplicity, interpretability, and usefulness, even in the age of big data. This article highlights the origins and developments of the boxplot that is now widely viewed as an industry standard as well as its inherent limitations when dealing with data from skewed distributions, particularly when detecting outliers. The proposed Ratio-Skewed boxplot is shown to be practical and suitable for outlier labeling across several parametric distributions.

An Efficient Incremental Density based Clustering Algorithm Fused with Noise Removal and Outlier Labelling Technique

Objectives: Due to advancements in storage technology, every moderate to large sized organization is keeping huge amount of multi facet data which is growing very fast. To deal with such enormous data, we need an efficient data analysis technique like classification and clustering. Methods/Statistical Analysis: Processing high dimensional data sets in the presence of noise and outliers can degrade the performance of any kind of data analysis task. The situation can even worse; if we are going for unsupervised classification (i.e. clustering).In this paper, we proposed a new method for incremental density based clustering for high dimensional data set with reasonable speed up. The proposed method fused with noise removal and outlier labeling technique is inspired from famous box plot method. Findings: The performance analysis of fusion is done on five high dimensional data sets taken from University California Irvine (UCI) repository along four cluster evaluation metrics (F-Measure, Entropy, Purity and Speed Up).The produced clustering results confirm the effectiveness of proposed fusion. Application/Improvements: The proposed technique can be refined by hybridizing it with some metaheuristic technique for stock exchange application.

Outlier Labeling Method for Univariate Data for Module Test and Die Sort

Qorvo is a global leader in scalable and dynamic RF solutions for mobile, infrastructure, and defense. To ensure quality and compliance to customer specifications, a large number of product devices and die on wafers are tested every day. Typical data sets will contain unusually large or small values when compared with others in that data set, which may or may not pass specification limits. Such outliers will have negative effects on data analyses such as cpk-analysis, ANOVA, and regressions. More important, outliers, particularly those passing specification limits, may indicate defective parts or pose a reliability concern. For engineering teams in development and production, it is standard practice to identify or label outliers in their data sets, so that they can be removed to improve data analyses and to protect customers against receiving defective parts. The aim for this paper is to provide product engineering teams at Qorvo with reliable outlier labeling methods particularly for skewed distributions. While the ultimate goal is to label and to remove outliers, we also need to balance this with unnecessary yield loss for the financial bottom line.

Simple outlier labeling based on quantile regression, with application to the steelmaking process

This paper introduces some methods for outlier identification in the regression setting, motivated by the analysis of steelmaking process data. The proposed methodology extends to the regression setting the boxplot rule, commonly used for outlier screening with univariate data. The focus here is on bivariate settings with a single covariate, but extensions are possible. The proposal is based on quantile regression, including an additional transformation parameter for selecting the best scale for linearity of the conditional quantiles. The resulting method is used to perform effective labeling of potential outliers, with a quite low computational complexity, allowing for simple implementation within statistical software as well as commonly used spreadsheets. Some simulation experiments have been carried out to study the swamping and masking properties of the proposal. The methodology is also illustrated by some real life examples, taking as the response variable the energy consumed in the melting process. Copyright © 2015 John Wiley & Sons, Ltd.

Note on the Comparison of Some Outlier Labeling Techniques

Problem statement: Methods proposed for estimating and resolving outliers are compared. Approach: In this respect, we exploit three well-known classifiers for identifying outliers to establish guidelines for the choice of outlier detection methods. Results: It was shown that the standard deviation is inappropriate to use here because it is highly sensitive to extreme values. Conclusion/Recommendations: The result of these estimated outliers is a better way of resolving large population.

Robust Multivariate Outlier Labeling

A criterion for robust estimation of location and covariance matrix is considered, and its application in outlier labeling is discussed. This method, unlike the methods based on MVE and MCD, is applicable to large and high-dimension data sets. The method proposed here is also robust and has the same breakdown point as the MVE- and MCD-based methods. Furthermore, the computational complexity of the proposed method is significantly smaller than that of other methods.

Resistant and Test-Based Outlier Rejection: Effects on Gaussian One- and Two-Sample Inference

Resistant and sequential test-based procedures for the detection of multiple outliers are compared in Gaussian one- and two-sample problems. Subsequent to statistical rejection of putative outliers, operating characteristics of hypothesis tests applied to samples that are outlier-free may differ from their nominal values. Outlier rejection rules may be calibrated, however, to control the rate of erroneous outlier labeling in pure Gaussian samples. The small-sample behavior of resistant location and scale estimators used in outlier detection is examined through simulation, yielding new formulas for calibration of boxplot-based and shorth-based outlier-rejection rules. Effects of different outlier criteria and calibration rubrics on postrejection hypothesis tests are compared in a variety of contamination settings and real-world examples. Tests conducted subsequent to calibrated rejection have approximately nominal operating characteristics whether or not outliers are present. The selection of an outlier cr...

Resistant and Test-Based Outlier Rejection: Effects on Gaussian One- and Two-Sample inference

Resistant and sequential test-based procedures for the detection of multiple outliers are compared in Gaussian one- and two-sample problems. Subsequent to statistical rejection of putative outliers, operating characteristics of hypothesis tests applied to samples that are outlier-free may differ from their nominal values. Outlier rejection rules may be calibrated, however, to control the rate of erroneous outlier labeling in pure Gaussian samples. The small-sample behavior of resistant location and scale estimators used in outlier detection is examined through simulation, yielding new formulas for calibration of boxplot-based and shorth-based outlier-rejection rules. Effects of different outlier criteria and calibration rubrics on postrejection hypothesis tests are compared in a variety of contamination settings and real-world examples. Tests conducted subsequent to calibrated rejection have approximately nominal operating characteristics whether or not outliers are present. The selection of an outlier criterion can have an impact on inference and thus becomes a potential component of model uncertainty. Properly calibrated tools should be widely available so that this component of uncertainty becomes better understood; we describe public-domain software implementing a variety of detection techniques.

Comparing Classical and Resistant Outlier Rules

Abstract In this article we compare Rosner's (1983) extension of the outlier t test (the generalized ESD procedure) to the boxplot rules, Tukey's (1977) resistant outlier labeling approach. The underlying principles are contrasted, and the behavior of the procedures is compared in a variety of sampling situations. To facilitate power-type comparisons of procedures that are based on different paradigms, the conditional outside rate, a performance criterion for assessing outlier labeling procedures, is introduced. This quantity represents the probability that an observation, yi , is labeled an outlier, conditional on its distance from the center, e = yi e μ. Some exact expressions, as well as simple approximations, and a Monte Carlo swindle are developed. The conditional outside rate, together with the outside rate per observation and the some-outside rate per sample, are applied to compare performance of the classical and boxplot rules, based on Monte Carlo simulation of samples of sizes n = 10, 30, and 50...

Some Asymptotic Analysis of Resistant Rules For Outlier Labeling

Previous studies have examined the behavior of outlier detection rules for symmetric distributions that label as “outside” any observations that fall outside the interval [FL – k(Fu – FL), Fu + k(Fu – FL)], where FL and FU are functions of the order statistics estimating the 0.25 and 0.75 quantiles of the distribution underlying the i.i.d. sample. A measure of the performance of this type of rule is the “some-outside rate” per sample computed with respect to a given (usually Gaussian) null distribution. The “some-outside rate” (SOR) per sample is the probability that the sample will contain one or more observations labeled as “outside,” given that the null distribution is the true distribution. In this paper, asymptotic expansions of k = kn as a function of n that guarantee an asymptotically constant, prespecified SOR are given for a variety of symmetric null distributions including the Gaussian, double exponential, logistic, and Cauchy distributions. The main theorem also applies to the case of a nonsymmetric null distribution by slightly modifying the labeling rule.

Fine-Tuning Some Resistant Rules for Outlier Labeling

Abstract A previous study examined the performance of a standard rule from Exploratory Data Analysis, which uses the sample fourths, FL and FU , and labels as “outside” any observations below FL – k(FU – FL ) or above FU + k(FU – FL ), customarily with k = 1.5. In terms of the order statistics X (1) ≤ X (2) ≤ X (n) the standard definition of the fourths is FL = X(f) and FU = X (n + 1 − f), where f = ½[(n + 3)/2] and [·] denotes the greatest-integer function. The results of that study suggest that finer interpolation for the fourths might yield smoother behavior in the face of varying sample size. In this article we show that using f i = n/4 + (5/12) to define the fourths produces the desired smoothness. Corresponding to a common definition of quartiles, fQ = n/4 + (1/4) leads to similar results. Instead of allowing the some-outside rate per sample (the probability that a sample contains one or more outside observations, analogous to the experimentwise error rate in simultaneous inference) to vary, some us...

Fine-Tuning Some Resistant Rules for Outlier Labeling

Abstract A previous study examined the performance of a standard rule from Exploratory Data Analysis, which uses the sample fourths, FL and FU , and labels as “outside” any observations below FL – k(FU – FL ) or above FU + k(FU – FL ), customarily with k = 1.5. In terms of the order statistics X (1) ≤ X (2) ≤ X (n) the standard definition of the fourths is FL = X(f) and FU = X (n + 1 − f), where f = ½[(n + 3)/2] and [·] denotes the greatest-integer function. The results of that study suggest that finer interpolation for the fourths might yield smoother behavior in the face of varying sample size. In this article we show that using f i = n/4 + (5/12) to define the fourths produces the desired smoothness. Corresponding to a common definition of quartiles, fQ = n/4 + (1/4) leads to similar results. Instead of allowing the some-outside rate per sample (the probability that a sample contains one or more outside observations, analogous to the experimentwise error rate in simultaneous inference) to vary, some users may prefer to maintain it at .10 or .05 for Gaussian data and vary k accordingly. We obtain such values of k at selected sample sizes n ≤ 300.

Performance of Some Resistant Rules for Outlier Labeling

Abstract The techniques of exploratory data analysis include a resistant rule for identifying possible outliers in univariate data. Using the lower and upper fourths, FL and FU (approximate quartiles), it labels as “outside” any observations below FL − 1.5(FU — FL ) or above FU + 1.5(FU — FL ). For example, in the ordered sample −5, −2, 0, 1, 8, FL = −2 and FU = 1, so any observation below −6.5 or above 5.5 is outside. Thus the rule labels 8 as outside. Some related rules also use cutoffs of the form FL — k(FU — FL ) and FU + k(FU — FL ). This approach avoids the need to specify the number of possible outliers in advance; as long as they are not too numerous, any outliers do not affect the location of the cutoffs. To describe the performance of these rules, we define the some-outside rate per sample as the probability that a sample will contain one or more outside observations. Its complement is the all-inside rate per sample. We also define the outside rate per observation as the average fraction of outside observations. For Gaussian data the population all-inside rate per sample (0) and the population outside rate per observation (.7%) substantially understate the corresponding small-sample values. Simulation studies using Gaussian samples with n between 5 and 300 yield detailed information on the resistant rules. The main resistant rule (k = 1.5) has an all-inside rate per sample between 67% and 86% for 5 ≤n ≤ 20, and corresponding estimates of its outside rate per observation range from 8.6% to 1.7%. Both characteristics vary with n in ways that lead to good empirical approximations. Because of the way in which the fourths are defined, the sample sizes separate into four classes, according to whether dividing n by 4 leaves a remainder of 0, 1, 2, or 3. Within these four classes the all-inside rate per sample shows a roughly linear decrease with n over the range 9 ≤ n ≤ 50, and the outside rate per observation decreases linearly in 1/n for n ≥ 9. A more theoretical approximation for the all-inside rate per sample works with the order statistics X (1) ≤ … ≤ X (n). In this notation the fourths are X(f) and X (n + 1 — f) with f = ½[(n + 3)/2], where [·] is the greatest-integer function. A sample has no observations outside whenever {X(f)−X(1)}/{X(n+1-f)−X(f)}≤k and {X(n)−X(n+1-f)}/{X(n+1-f)−X(f)}≤k. We first approximate the numerators and denominator in these ratios by constant multiples of chi-squared random variables with the same mean and variance. We then approximate the logarithm of each ratio by a Gaussian random variable, and we calculate the correlation between these variables from the fact that the ratios have the same denominator. Finally, a bivariate Gaussian probability calculation yields the approximate all-inside rate per sample. The error of the result relative to the simulation estimate is typically from 1% to 2% for 5 ≤ n ≤ 50. To provide an indication of how the two rates behave in alternative “null” situations, the simulation studies included samples from five heavier-tailed members of the family of h distributions. For a given sample size, the all-inside rate per sample decreases as the tails become heavier, and the outside rate per observation increases.

Performance of Some Resistant Rules for Outlier Labeling

Abstract The techniques of exploratory data analysis include a resistant rule for identifying possible outliers in univariate data. Using the lower and upper fourths, FL and FU (approximate quartiles), it labels as “outside” any observations below FL − 1.5(FU — FL ) or above FU + 1.5(FU — FL ). For example, in the ordered sample −5, −2, 0, 1, 8, FL = −2 and FU = 1, so any observation below −6.5 or above 5.5 is outside. Thus the rule labels 8 as outside. Some related rules also use cutoffs of the form FL — k(FU — FL ) and FU + k(FU — FL ). This approach avoids the need to specify the number of possible outliers in advance; as long as they are not too numerous, any outliers do not affect the location of the cutoffs. To describe the performance of these rules, we define the some-outside rate per sample as the probability that a sample will contain one or more outside observations. Its complement is the all-inside rate per sample. We also define the outside rate per observation as the average fraction of outs...