Instruction Matters: Lessons from a Mixed-Method Evaluation of Out-Of-School Time Tutoring under No Child Left Behind

Abstract

Background/Context Under supplemental educational services (“supplemental services”), a parental choice provision of the No Child Left Behind Act (NCLB), schools that have not made adequate yearly progress in increasing student achievement are required to offer low-income families free, afterschool tutoring. Existing research shows low attendance rates among eligible students and little to no aggregate effects on achievement for students who do attend. Focus of study We employ a framework grounded in examining the instructional setting, or “instructional core,” and we draw on the unique contributions of qualitative research to help explain the limited effects of supplemental services on student achievement. Specifically, we address the following research question: How can in-depth examination of the instructional core explain the impact of supplemental services on student learning? Research Design Our findings draw on data from an ongoing mixed-method and multisite study of the implementation and impact of supplemental educational services in five urban school districts located in four states. Although this paper includes quantitative data from this study, analysis focuses on qualitative data, including observations of tutoring sessions using a standardized observation instrument; semistructured interviews with district staff, provider administrators, and tutors; focus groups with parents of eligible students; and document analysis. Findings We identify two primary reasons for a lack of effects. First, there is a “treatment exposure” problem where most students receive far less than 40 hours of tutoring over the course of a school year, a critical threshold for seeing significant effects on achievement. In addition, there are discrepancies between an invoiced hour of tutoring and actual instructional time. Second, supplemental services has an instructional quality problem. Instruction lacks innovation; the curriculum typically does not align to that of the day school; programs do not meet all students’ instructional needs, especially students with disabilities and English language learners; and there can be considerable variation in quality within the same provider. Conclusions Our findings lay the foundations for being able to not only establish best practices for supplemental services, but to suggest policy changes to facilitate these best practices and offer insights to a host of other parental choice, out-of-school time (OST), and accountability-based reforms.