Outlier Detection Rules Research Articles

Iterative outlier detection and refinement rule of compensation for phase aberrations in digital holographic microscopy.

We propose a robust and accurate compensation method for phase aberrations based on the iterative outlier detection and refinement (ODR) rule. This method does not require additional steps to select the known flat region manually or by image segmentation. Based on the proposed method, the phase aberration in regions of a specimen can be detected and refined iteratively. Then, the least squares fitting can be carried out to estimate the coefficients of Zernike polynomials and obtain the accurate phase aberration finally. Computer simulations and real experiments validate the feasibility and effectiveness, and the results show that the proposed method is robust to noise and has superior accuracy even when the specimen occupies half of the field of view.

Robust estimation for bivariate integer-valued autoregressive models based on minimum density power divergence

This study considers a robust estimation method for bivariate integer-valued autoregressive (BINAR) models using a minimum density power divergence (MDPD) scheme. We first address this problem within a general framework of time series models to verify the strong consistency and asymptotic normality of the MDPD estimator (MDPDE) under regularity conditions. Based on these results, we assert the same asymptotic properties of MDPDE for BINAR models and propose an outlier detection rule based on MDPDE. To assess the performance of the proposed method, we conduct a simulation study. Real data analysis is also carried out for illustration, using the number of monthly earthquakes in the United States.

Analysis of outlier detection rules based on the ASHRAE global thermal comfort database

Analysis of outlier detection rules based on the ASHRAE global thermal comfort database

Robust Fitting of a Wrapped Normal Model to Multivariate Circular Data and Outlier Detection

In this work, we deal with a robust fitting of a wrapped normal model to multivariate circular data. Robust estimation is supposed to mitigate the adverse effects of outliers on inference. Furthermore, the use of a proper robust method leads to the definition of effective outlier detection rules. Robust fitting is achieved by a suitable modification of a classification-expectation-maximization algorithm that has been developed to perform a maximum likelihood estimation of the parameters of a multivariate wrapped normal distribution. The modification concerns the use of complete-data estimating equations that involve a set of data dependent weights aimed to downweight the effect of possible outliers. Several robust techniques are considered to define weights. The finite sample behavior of the resulting proposed methods is investigated by some numerical studies and real data examples.

Outlier Detection Based on Multivariable Panel Data and K‐Means Clustering for Dam Deformation Monitoring Data

A dam is a super‐structure widely used in water conservancy engineering fields, and its long‐term safety is a focus of social concern. Deformation is a crucial evaluation index and comprehensive reflection of the structural state of dams, and thus there are many research papers on dam deformation data analysis. However, the accuracy of deformation data is the premise of dam safety monitoring analysis, and original deformation data may have some outliers caused by manual errors or instruments aging after long‐time running. These abnormal data have a negative impact on the evaluation of dam structural safety. In this study, an analytical method for detecting outliers of dam deformation data was established based on multivariable panel data and K‐means clustering theory. First, we arranged the original spatiotemporal monitoring data into the multivariable panel data format. Second, the correlation coefficients between the deformation signals of different measuring points were studied based on K‐means clustering theory. Third, the outlier detection rules were established through the changes of the correlation coefficients. Finally, the proposed model was applied to the Jinping‐I Arch Dam in China which is the highest dam in the world, and results indicate that the detection method has high accuracy detection ability, which is valuable in dam safety monitoring applications.

Performance Improvement of Gene Selection Methods using Outlier Modification Rule

Background: DNA microarray technology allows researchers to measure the expression levels of thousands of genes simultaneously. The main objective of microarray gene expression (GE) data analysis is to detect biomarker genes that are Differentially Expressed (DE) between two or more experimental groups/conditions. Objective: There are some popular statistical methods in the literature for the selection of biomarker genes. However, most of them often produce misleading results in presence of outliers. Therefore, in this study, we introduce a robust approach to overcome the problems of classical methods. Methods: We use median and median absolute deviation (MAD) for our robust procedure. In this procedure, a gene was considered as outlying gene if at least one of the expressions of this gene does not belong to a certain interval of the proposed outlier detection rule. Otherwise, this gene was considered as a non-outlying gene. Results: We investigate the performance of the proposed method in a comparison of the traditional method using both simulated and real gene expression data analysis. From a real colon cancer gene expression data analysis, the proposed method detected an additional fourteen (14) DE genes that were not detected by the traditional methods. Using the Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway enrichment analysis, we observed that these additional 14 DE genes are involved in three important metabolic pathways of cancer disease. The proposed method also detected nine (9) additional DE genes from another head-and-neck cancer gene expression data analysis; those involved in top ten metabolic pathways obtain from the KEGG pathway database. Conclusion: The simulation as well as real cancer gene expression datasets results show better performance with our proposed procedure. Therefore, the additional genes detected by the proposed procedure require further wet lab validation.

Weighted likelihood estimation of multivariate location and scatter

A novel approach to obtain weighted likelihood estimates of multivariate location and scatter is discussed. A weighting scheme is proposed that is based on the univariate distribution of the Mahalanobis distances rather than the multivariate distribution of the data at the assumed model. This strategy allows to avoid the curse of dimensionality affecting multivariate non-parametric density estimation, that is involved in the construction of the weights through the Pearson residuals. Asymptotic properties of the proposed weighted likelihood estimator are also discussed. Then, weighted likelihood-based outlier detection rules and robust dimensionality reduction techniques are developed. The effectiveness of the methodology is illustrated through some numerical studies and real data examples.

ICS for multivariate outlier detection with application to quality control

In high reliability standards fields such as automotive, avionics or aerospace, the detection of anomalies is crucial. An efficient methodology for automatically detecting multivariate outliers is introduced. It takes advantage of the remarkable properties of the Invariant Coordinate Selection (ICS) method which leads to an affine invariant coordinate system in which the Euclidian distance corresponds to a Mahalanobis Distance (MD) in the original coordinates. The limitations of MD are highlighted using theoretical arguments in a context where the dimension of the data is large. Owing to the resulting dimension reduction, ICS is expected to improve the power of outlier detection rules such as MD-based criteria. The paper includes practical guidelines for using ICS in the context of a small proportion of outliers. The use of the regular covariance matrix and the so called matrix of fourth moments as the scatter pair is recommended. This choice combines the simplicity of implementation together with the possibility to derive theoretical results. The selection of relevant invariant components through parallel analysis and normality tests is addressed. A simulation study confirms the good properties of the proposal and provides a comparison with Principal Component Analysis and MD. The performance of the proposal is also evaluated on two real data sets using a user-friendly R package accompanying the paper.

Multiple outliers detection in sparse high-dimensional regression

ABSTRACTThe presence of outliers would inevitably lead to distorted analysis and inappropriate prediction, especially for multiple outliers in high-dimensional regression, where the high dimensionality of the data might amplify the chance of an observation or multiple observations being outlying. Noting that the detection of outliers is not only necessary but also important in high-dimensional regression analysis, we, in this paper, propose a feasible outlier detection approach in sparse high-dimensional linear regression model. Firstly, we search a clean subset by use of the sure independence screening method and the least trimmed square regression estimates. Then, we define a high-dimensional outlier detection measure and propose a multiple outliers detection approach through multiple testing procedures. In addition, to enhance efficiency, we refine the outlier detection rule after obtaining a relatively reliable non-outlier subset based on the initial detection approach. By comparison studies based on Monte Carlo simulation, it is shown that the proposed method performs well for detecting multiple outliers in sparse high-dimensional linear regression model. We further illustrate the application of the proposed method by empirical analysis of a real-life protein and gene expression data.

To Remove or not to Remove: the Impact of Outlier Handling on Significance Testing in Testosterone Data

Outlier removal is common in hormonal research. Here we investigated to what extent removing outliers in hormonal data leads to divergent statistical conclusions. We first show that the most common outlier detection rule is based on a number of standard deviations (SD) from the mean. Next, we used simulations to examine the degree to which statistical conclusions diverge when a test with outlier exclusion yields a statistically significant result whereas the test with outlier inclusion did not, or vice versa (at p = .05). Simulations were run in duplicate for independent samples t-tests and repeated measures ANOVA designs, and based on real testosterone (T) data and a theoretical gamma distribution of T data. We ran simulations for different sample sizes (30 to 100) and outlier removal rules (2.5 SD and 3 SD). For significant t-tests, we found that in between 14 % to 55 % of the significant cases a test with outlier exclusion yielded a statistically significant result whereas the test with outlier inclusion did not, or vice versa (median p difference: .03–.06). For significant repeated measures ANOVAs, we found that in between 7 % to 28 % of significant cases a test where outlier exclusion yielded a statistically significant result whereas the test with outlier inclusion did not, or vice versa (median p difference: .01–.03). When reporting any test that would lead to a statistically significant result (either the test with inclusion or exclusion of outliers (or both)), in between 5.15 % and 6.89 % of the independent sample t-tests were statistically significant, and for the repeated measures ANOVA design this was between 6.32 % and 7.62 % of the tests. Our results suggest that outlier handling can have a substantial impact on significance testing. We suggest several potential solutions for handling outliers and we argue for a careful assessment of handling outliers in hormonal data.

Comments on: Robust estimation of multivariate location and scatter in the presence of cellwise and casewise contamination

Agostinelli, Leung, Yohai and Zamar (Agostinelli et al. in the remainder) consider the difficult problem of robust estimation based on high-dimensional data. If outlying values can appear independently in the variables, then it can easily occur that the majority of the observations in high-dimensional data is contaminated, as pointed out in Alqallaf et al (2009). Consequently, standard robust methods fail in this case, and new methods need to be developed that can handle this type of contamination. Moreover, next to independent contamination also casewise or structural outliers can still appear in the data. This situation was formalized as the partially spoiled independent contamination model (PSICM) in Alqallaf et al (2009). In their paper Agostinelli et al. are the first to introduce a consistent estimator of multivariate location and scatter that is highly robust against both cellwise and casewise outliers. The 2SGS is a strongly consistent estimator of the location and shape of general elliptical distributions. Similarly to other proposals, the estimator proceeds in two steps. In the first step an outlier detection rule is used to identify potential cellwise outliers. A first improvement is the use of a data adaptive cutoff instead of a fixed cutoff value when filtering cellwise outliers. The second novelty is to replace flagged outliers by missing values as first proposed in Danilov (2010) and Farcomeni (2013), while earlier proposals tried to reduce their effect through some form of Winsorization, see e.g. Alqallaf et al. (2002), Van Aelst et al. (2011,2012), Van Aelst (2015). In the second step, the location and scatter are estimated based on the data set with missing values by using the GSE estimator of Danilov et al (2012). GSE

Boxplot-Based Outlier Detection for the Location-Scale Family

Boxplots are among the most widely used exploratory data analysis (EDA) tools in statistical practice. Typical applications of boxplots include eliciting information about the underlying distribution (shape, location, etc.) as well as identifying possible outliers. This article focuses on a modification using a type of lower and upper fences similar in concept to those used in a traditional boxplot; however, instead of constructing the upper and lower fences using the upper and lower quartiles, respectively, and a multiple of the interquartile range (IQR), multiples of the upper and the lower semi-interquartile ranges (SIQR), respectively, measured from the sample median, are used. Any observation beyond the proposed fences is labeled a potential outlier. An exact expression for the probability that at least one sample observation is wrongly classified as an outlier, the so-called “some-outside rate per sample” (Hoaglin et al. (1986)), is derived for the family of location-scale distributions and is used in the determination of the fence constants. Tables for the fence constants are provided for a number of well-known location-scale distributions along with some illustrations with data; the performance of the outlier detection rule is explored in a simulation study.

Resistant estimators in Poisson and Gamma models with missing responses and an application to outlier detection

When dealing with situations in which the responses are discrete or show some type of asymmetry, the linear model is not appropriate to establish the relation between the responses and the covariates. Generalized linear models serve this purpose, since they allow one to model the mean of the responses through a link function, linearly on the covariates. When atypical observations are present in the sample, robust estimators are useful to provide fair estimations as well as to build outlier detection rules. The focus of this paper is to define robust estimators for the regression parameter when missing data possibly occur in the responses. The estimators introduced turn out to be consistent under mild conditions. In particular, resistant methods for Poisson and Gamma models are given. A simulation study allows one to compare the behaviour of the classical and robust estimators, under different contamination schemes. The robustness of the proposed procedures is studied through the influence function, while asymptotic variances are derived from it. Besides, outlier detection rules are defined using the influence function. The procedure is also illustrated by analysing a real data set.

Adjusted functional boxplots for spatio‐temporal data visualization and outlier detection

This article proposes a simulation‐based method to adjust functional boxplots for correlations when visualizing functional and spatio‐temporal data, as well as detecting outliers. We start by investigating the relationship between the spatio‐temporal dependence and the 1.5 times the 50% central region empirical outlier detection rule. Then, we propose to simulate observations without outliers on the basis of a robust estimator of the covariance function of the data. We select the constant factor in the functional boxplot to control the probability of correctly detecting no outliers. Finally, we apply the selected factor to the functional boxplot of the original data. As applications, the factor selection procedure and the adjusted functional boxplots are demonstrated on sea surface temperatures, spatio‐temporal precipitation and general circulation model (GCM) data. The outlier detection performance is also compared before and after the factor adjustment. Copyright © 2011 John Wiley & Sons, Ltd.

Rebooting the human mitochondrial phylogeny: an automated and scalable methodology with expert knowledge.

BackgroundMitochondrial DNA is an ideal source of information to conduct evolutionary and phylogenetic studies due to its extraordinary properties and abundance. Many insights can be gained from these, including but not limited to screening genetic variation to identify potentially deleterious mutations. However, such advances require efficient solutions to very difficult computational problems, a need that is hampered by the very plenty of data that confers strength to the analysis.ResultsWe develop a systematic, automated methodology to overcome these difficulties, building from readily available, public sequence databases to high-quality alignments and phylogenetic trees. Within each stage in an autonomous workflow, outputs are carefully evaluated and outlier detection rules defined to integrate expert knowledge and automated curation, hence avoiding the manual bottleneck found in past approaches to the problem. Using these techniques, we have performed exhaustive updates to the human mitochondrial phylogeny, illustrating the power and computational scalability of our approach, and we have conducted some initial analyses on the resulting phylogenies.ConclusionsThe problem at hand demands careful definition of inputs and adequate algorithmic treatment for its solutions to be realistic and useful. It is possible to define formal rules to address the former requirement by refining inputs directly and through their combination as outputs, and the latter are also of help to ascertain the performance of chosen algorithms. Rules can exploit known or inferred properties of datasets to simplify inputs through partitioning, therefore cutting computational costs and affording work on rapidly growing, otherwise intractable datasets. Although expert guidance may be necessary to assist the learning process, low-risk results can be fully automated and have proved themselves convenient and valuable.

Adaptive trimmed t‐statistics for identifying predominantly high expression in a microarray experiment

Often, interesting candidate tumor markers are not only genes that show homogeneously higher expression (HHE) in tumor samples compared to control samples, but also genes with only predominantly higher expression (PHE), i.e. genes which exhibit higher expression in at least 80 per cent of tumor samples. Standard parametric test statistics used in the analysis of microarray experiments may fail with PHE as a consequence of the mixture of distributions present in the tumor group. As alternative we consider trimmed t-statistics which compare group mean values after removing outliers in each group. The trimming proportion can be chosen adaptively, either based on a boxplot outlier detection rule or by optimization over a series of tests with varying trimming proportions. The trimmed t-statistics can be plugged into the 'significance analysis of microarrays' (SAM) procedure, yielding the modified boxplot rule test (modBox) and the modified optimization test (modOpt), respectively. By means of simulation of microarray experiments, we show that modOpt is superior to contenders in detecting PHE, while there is only little loss in efficiency under HHE compared to SAM. Analysis of a real microarray experiment revealed that, out of nearly 29 000 genes, about 417 genes exhibiting PHE are detected by modOpt but missed by SAM.

Error rates for multivariate outlier detection

Multivariate outlier identification requires the choice of reliable cut-off points for the robust distances that measure the discrepancy from the fit provided by high-breakdown estimators of location and scatter. Multiplicity issues affect the identification of the appropriate cut-off points. It is described how a careful choice of the error rate which is controlled during the outlier detection process can yield a good compromise between high power and low swamping, when alternatives to the Family Wise Error Rate are considered. Multivariate outlier detection rules based on the False Discovery Rate and the False Discovery Exceedance criteria are proposed. The properties of these rules are evaluated through simulation. The rules are then applied to real data examples. The conclusion is that the proposed approach provides a sensible strategy in many situations of practical interest.

Outlier detection for skewed data

AbstractMost outlier detection rules for multivariate data are based on the assumption of elliptical symmetry of the underlying distribution. We propose an outlier detection method which does not need the assumption of symmetry and does not rely on visual inspection. Our method is a generalization of the Stahel–Donoho outlyingness. The latter approach assigns to each observation a measure of outlyingness, which is obtained by projection pursuit techniques that only use univariate robust measures of location and scale. To allow skewness in the data, we adjust this measure of outlyingness by using a robust measure of skewness as well. The observations corresponding to an outlying value of the adjusted outlyingness (AO) are then considered as outliers. For bivariate data, our approach leads to two graphical representations. The first one is a contour plot of the AO values. We also construct an extension of the boxplot for bivariate data, in the spirit of the bagplot 1 which is based on the concept of half space depth. We illustrate our outlier detection method on several simulated and real data. Copyright © 2008 John Wiley & Sons, Ltd.

Statistical algorithm for assuring similar efficiency in standards and samples for absolute quantification by real-time reverse transcription polymerase chain reaction

Reverse transcription (RT) followed by the polymerase chain reaction (PCR) is the method of choice for quantifying rare transcripts in biological samples. A key assumption underlying the absolute quantification of transcripts is similar amplification efficiencies of all external standards and samples. However, efficiencies can vary between individual reactions, a problem that can be magnified when quantifying transcripts of low abundance. Here, an algorithm to assure that calibration standards and samples meet the assumption of similar amplification efficiencies underlying absolute quantification is presented. Individual reaction efficiency is estimated by fitting an exponential growth model to the fluorescence data in the exponential phase of the reaction. Next, reactions of standards with outlying estimates of amplification rates are eliminated using the boxplot outlier detection rule. Then, estimates of amplification rates of outlier-free standards are employed to define exact tolerance intervals, which are used to eliminate kinetic outliers from test samples. This algorithm was employed to eliminate kinetic outliers prior to defining the baseline expression of guanylyl cyclase C mRNA, a marker for colorectal cancer, in blood of healthy volunteers. These studies demonstrate that elimination of kinetic outliers from calibration standards and test samples improves the accuracy of absolute transcript quantification by RT-PCR.

Outlier detection and trimmed means for poisson data

Abstract In some situations a small proportion of data do not arise from a given Poisson distribution but from a Poisson distribution with a different parameter value. The non-Bayesian and Bayesian outlier detection rules are proposed based on the square root transformed data. Different trimmed means are formed according to the above mentioned outlier detection methods. The relative efficiencies of these estimators with respect to the sample mean, and the mean proportions of data classified as outliers are computed by using Monte Carlo methods. Based on the computer simulation we can conclude that the non-Bayesian estimators are good robust estimators.