Methodology for detecting and removing outliers in statistical studies

Abstract

The paper presents a calculation method for detecting and eliminating outlying values. It is shown that its effectiveness depends on the amount of a priori information on the examined process. The proposed method is used for cases whereas the process is stationary and has a Gaussian probability density law. When analysing non-stationary random processes, the existing methods and algorithms rely on the fact that the outlying component is additive and the characteristics of the outlying values are known a priori. The work used the statistical decisions theory that allows formalising the verification algorithms and selecting a criterion for detecting outlying values. Both parametric and non-parametric methods were proposed. In the first case, it is required to have a priori information both on the function of the useful component and on the distribution law of the outlying component of the process, as well as its parameters. It is postulated that the use of non-parametric processing methods requires significantly less a priori information, but their effectiveness is defined by the processing parameters that, in turn, depend on the function of the useful and the distribution law of the outlying components of the process. It is noted that an outlier may prove to be one of the extreme values of the probability distribution of a random variable. The authors outline the problems of ambiguity of input data in case of classical computing. The paper examines the way the external factors affect the dependability and the degree to which such factors are taken into consideration in the existing methods. Methods for assessing the life of the examined items are presented, among which control chart-based methods hold a prominent place. It is shown that the range proves to be a more convenient measure for data dispersion calculation than the standard deviation. Plotting the range of sample on a control chart along with the expectation makes it easier to notice an anomaly. The range is a rough measure of the rate of change of the monitored variable and its value may exceed the control limits on the range chart and inform of an anomaly much earlier than the change in the mean that may still be within the specified control limits.