Measuring Textbook Content Coverage: Efficient Content Analysis with Lesson Sampling

Abstract

AbstractThis study addresses measurement issues around a standards‐based content analysis of mathematics textbooks’ coverage of standards for use in large‐scale monitoring of standards implementation as proposed in a 2013 report by the National Research Council. An earlier study produced an exhaustive content analysis of textbooks using the 2012 Common Core State Standards for Mathematics. This yielded an accurate and reliable portrait of a textbook's coverage of standards. However, such an in‐depth analysis is not feasible for large‐scale standards‐implementation monitoring in which a large number of textbooks may need to be analyzed. To provide such a portrait with sufficient accuracy while also substantially reducing the associated resources needed to produce such a portrait, a simulation study was conducted with the exhaustively coded database to compare different sampling schemes. Results indicated that sampling 1 day from each week and coding the corresponding lessons led to sufficiently accurate representations of the overall content of the textbook. The results provide empirical evidence for large‐scale standards‐based content analyses of mathematics textbooks for monitoring standards implementation which could be adapted for other subject areas.