Sensitivity of Item Difficulties to Curricular Validity

Abstract

The issue of the match of what is taught to what is tested recently has received increased emphasis. Many recent studies have examined the extent of the mismatch between a local set of objectives/instruction and the content of a nationally standardized test. Phillips and Mehrens (1985, 1987; see also Mehrens & Phillips, 1986) have conducted a series of studies that show that, within a district, different textbook series and informal curricula generally have no significant impact on test total, objective, or item scores. The present study explored further the curricular validity differences across textbooks and the relationship between item statistics and measures of curricular validity. In particular, this study sought to determine whether item p‐values appear to be related to measures of curricular validity based on three mathematics textbook series used at a given grade and in a previous grade. The results of this study indicated that the textbooks differed somewhat in content coverage when using a 180‐cell matrix classification. However, these differences were not great, especially when the textbooks in both grades 5 and 6 were considered. Further, all three series covered almost all of the 53 cells in the matrix covered by the Stanford Achievement Test, and the differences that did exist in textbook content coverage had no observable relationship to differences in item p‐values. In addition, the mean difficulty level of the Stanford Achievement Test items classified by cell were similar for students using the different textbooks, despite the differences among textbooks in location, presentation, and organization of content.