Assessing the consequential validity of early literacy progress monitoring data: An investigation of the accuracy of decision rules to evaluate response to instruction.

Abstract

A growing body of research suggests that growth in early literacy skills, including letter sound fluency, is predictive of later reading outcomes. In turn, educators and school psychologists use measures of letter sound fluency to monitor student response to early reading instruction. Limited research has evaluated whether decision-making frameworks that educators apply to early reading time series data to make instructional decisions (e.g., to continue or change the intervention) yield accurate recommendations. Further, it is unclear how long data need to be collected and which type of decision rule (e.g., data point, trend line) will produce the most accurate recommendations. We conducted a series of simulations to investigate the impact of data collection duration (4 to 16 weeks) and decision-rule type (data point, trend line, and median) on the accuracy of data-based decisions using early literacy data. Results suggest that the median and trend-line rules produced recommendations that were sufficiently accurate in identifying students that were not making adequate progress (sensitivity) after about 12 weeks when data were collected once a week. However, the median and trend-line rules did not produce recommendations that were sufficiently accurate to continue an intervention (specificity) across all progress monitoring durations. The opposite pattern was observed for the data-point rule. Outcomes suggest that recommendations developed from other progress monitoring measures (e.g., oral reading fluency) should not be extrapolated to other measures without empirical investigation. More research is needed to identify appropriate decision rules to evaluate early literacy progress monitoring data. (PsycINFO Database Record (c) 2019 APA, all rights reserved).