On comparing different tests of the same hypothesis

Abstract

a series of small samples of real data has been examined by Pearson & Haines (1935) who plotted w against s in a control chart situation. With the help of the known standard deviation cr they were able to insert on their diagrams a grid of several upper- and lowerpercentage points of bothwands so that the corresponding degrees of significance attained by any sample and the agreement in this respect could be seen on inspection. A similar approach, but based on experimental sampling from normal populations, was used by Eisenhart & Swed (1940) to compare two possible tests of the homogeneity of several estimates of variance. In the null case they found considerable disagreement at the 5 % level of significance. More recently Cox (1956) has discussed some general aspects of this problem. We now restate and add to some of his arguments: (a) The case for a particular quick test is strengthened if it is known that it will nearly always lead to the same action as the standard test. (b) Quick tests are often used in preliminary or spot analyses. It seems highly desirable that the conclusions so reached should not be seriously upset if a fuller analysis is carried out. The quick test should inspire confidence in this respect even if it will not be followed up at all. (c) The method of data collection or experimentation may lend itself to a particularly advantageous use of quick tests; indeed, the experimental approach may be modified for this reason. To be specific let us consider a life test on items with a normal distribution, N(0, cr2), of lives. The test may be stopped as soon as just over half the items in it have failed. If cr2 is assumed known a very simple test on 0 may be based on the sample median. One could, of course, improve the analysis by using all the available observations and not only the median. More fundamentally, the experimental method hinges on the fact that it allows a quick decision to be reached. Confidence in the reliability of this approach is presumably based on the knowledge that the median leads to a reasonably powerful test, but also on the supposition that the conclusion reached with the censored test would generally be confirmed were the test in fact run to completion. A similar point could be made in comparing the paired t-test and the sign test: not only does the sign test rest on fewer assumptions, it also allows a simplification of experimental technique.