A Comparison of Reliability Estimation Using Trials-to-Criterion and Sequential Probability Ratio Testing

Abstract

The purpose of this study was to compare estimates of test reliability obtained from two sequential testing plans--trials-to-criterion (TTC) and sequential probability ratio (SPR) testing--when reliability is defined as the consistency of classification. Data from a golf chip test given to 110 beginning golf students (n = 80 males; n = 30 females) at the University of Wisconsin were used for analysis. Test specifications for the SPR test were alpha = beta = .05, theta 0 = .70, and theta 1 = .50. Two mastery levels for the TTC test were examined, .70 and .60, with success criteria ranging from R = 6 to R = 12. For each sequential testing plan, both P and kappa were calculated to estimate reliability. Results for the total group and for gender indicated that reliability was higher with the SPR test when the mastery level was .70, while reliability was similar under both plans at a mastery level of .60. Median test lengths for the group were 21 for the SPR test and an average of 12 across all R values for the TTC test. Misclassification error rates for the TTC test, however, were substantially higher than under the SPR test, particularly for false nonmaster errors. These data suggest that SPR testing would be the preferred approach when misclassification errors are of primary importance, such as to determine minimal competency for certification. However, TTC testing is a viable alternative for classroom tests because of ease of administration and shorter test length.