Subjective Judgment in Statistical Analysis: An Experimental Study

Abstract

Summary As part of an experimental study of the effect of stopping iterative maximum likelihood calculations after one cycle, twenty-one scientists were asked to judge by eye the positions for a pair of parallel probit regression lines in a biological assay. This paper reports the results obtained from each pair of lines after one cycle of various forms of probit calculation. Although all the conditions of the experiment were chosen so as to be unfavourable to the drawing of good lines, one cycle of iteration brought every estimate within 1½ per cent. of the maximum likelihood value, as compared with a range of 32 per cent. and a maximum error of 27 per cent. for the estimates read directly from the graphs. The fiducial limits as calculated after one cycle were brought within a range of about 5 per cent. around the true values. The subjects for this study were deliberately chosen as having no experience of probit methods. The results confirm me in the opinion I have often expressed, namely that, unless a diagram showing probits plotted against log dose is exceptionally irregular, a single cycle of iteration initiated by any reasonable trial regression line will give a satisfactory approximation. Experience in drawing the trial lines undoubtedly makes the first cycle better still, but even the inexperienced can do quite well. I am in no way suggesting that the critical faculty of the statistician is unnecessary in deciding when iteration may be stopped: care is needed in order to ensure that all doubtful cases have additional cycles, but users of the probit method may find some comfort in the thought that their trial lines are seldom as bad as they fear! When the form of calculation recently proposed by Cornfield and Mantel was used, results after one cycle were rather better, but, since those by the usual technique were so good, the practical advantage was not great. Additional calculations on some very extreme pairs of trial lines suggest that, if the data were so scanty or so irregular as to make the positions of the lines very uncertain, the Cornfield-Mantel procedure would then be markedly superior: in such circumstances, more than one cycle of iteration would certainly be needed, but this procedure might reduce the number of cycles. A further practical point in connection with data of this type is that an error of making the slopes of the trial lines too small is preferable to an error of making the slopes too large, since the latter is more likely to increase the number of cycles needed or even to make the iteration diverge.