An Investigation of Ordinal True Score Test Theory

Abstract

The validity of the assumptions underlying Cliff's (1989) ordinal true score theory (OTST) were investigated in a three-stage study. OTST makes only ordinal assumptions about the data, and provides a means of converting ordinal item information into summary ordinal information about examinees. Stage 1 was a simulation based on a classical (weak true score) test theory model. Stage 2 used a long empirical test to approximate the true order. Stage 3 was an extensive simulation based on the three-parameter logistic model. The results of all three studies were consistent; the assumption of local ordinal uncorrelatedness was violated in that partial item-item gamma (y) cor relations were positive instead of 0. The assump tion of proportional distribution of ties was violated—pairs tied on one item were not distributed on the other as prescribed. The item- true order tau (τ) correlation was consistently overestimated, although the estimated τ correlated highly with the true τ. The τ correlation between total score and true order was also consistently overestimated. Stage 3 showed that these effects occurred under all conditions, although they were smaller under some conditions.