Estimating the Reliability of Dichotomous or Trichotomous Scores

Abstract

To meet the requirements of the No Child Left Behind Act, school districts and states must compile summary reports of the levels of student achievement in reading and mathematics. The levels are to be described in broad categories: “basic and below,” “proficient,” or “advanced.” Educational units are given considerable latitude in defining the percentile limits of these categories and the measures to be used in assessing the levels of each student. However, they must defend the validity and reliability of these measures. The question arises: How does one estimate the reliability of a score generated by a test of k items after the score is reduced to three or even two values? A precise solution, based on normality assumptions, requires the application of computer programs inaccessible to most local educators. Tables are provided in this article that enable local educators to deduce the reliability of broad-category scores (ρ yy′) from the reliability of the original scores (ρ xx′).