The reliability ritual

Abstract

A critical appraisal is presented of some of the misconceptions surrounding the notion of test reliability which appear to be quite pervasive in some linguistic research circles. First, it is demonstrated that observed group differences which prove significant in the adequate significance tests, are in no way discredited by low reliability coefficients. Second, it is argued that test reliability should be determined a priori, i.e. before a test is actually used in an investigation. Post-hoc computations of test reliability are redundant and may even be misleading. Third, the results of computer simulations are presented which show that reliability coefficients such as Cronbach's alpha depend to a large degree on the heterogeneity of the subject sample and the range of item dif ficulties. This imposes serious limitations on the usefulness of Cronbach's alpha as a reliability measure, especially when computed post-hoc. As a scalability measure Cronbach's alpha seems fundamentally inappropriate.