Masking effect on tests for outliers in normal samples

Abstract

Many tests have been proposed for detecting a preassigned number, k 1, of outliers in a sample of size n assumed to be from a normal population. Barnett & Lewis (1984, ? 6.3) list most of these and also discuss their rationale. Pearson & Chandrasekar (1936), Dixon (1950) and Grubbs (1950) had already observed that, if we are testing for a single outlier and if in fact there is more than one outlier, say two, present, then the test for a single outlier would fail to detect even a single outlier. This phenomenon was termed a masking effect by R. B. Murphy in his Princeton Ph.D. thesis. The masking effect has been discussed widely in the literature although mostly by examples. Bendre & Kale (1985) introduced a simple measure of the masking effect as the loss in power due to the presence of more than the number of outliers under test, and studied this for the test for a single outlier in samples from an exponential distribution. The object of the present paper is to use the same measure on some of the commonly applied tests for detecting outliers in samples from a normal distribution. We illustrate the technique by first considering Grubbs's test for a single outlier, which is equivalent to the test based on the maximum studentized deviation first proposed by Pearson & Chandrasekar (1936), and then briefly indicating how the technique can be applied to tests based on skewness proposed by Ferguson (1961) as a locally most powerful invariant test and to Dixon's (1950) test. We conclude that Ferguson's test is much less vulnerable to a masking effect, whereas Dixon's test is highly susceptible, with Grubbs's test falling between the two. As pointed out by a referee, this conclusion is based on a single measure of masking effect and the Dixon test, even though suffering from the masking effect, could be recommended on other grounds. While only three tests are considered, the techniques developed here enable study of the masking effect on most of the tests used for detecting outliers in normal samples.