Measuring the Average Impact of an iPad Algebra Program A Report of Findings from an RCT in Four School Districts Considering One as a Special Case

Abstract

Introduction: In spring 2010, Houghton Mifflin Harcourt (HMH) began planning a pilot of an application for the Apple iPad, HMH Fuse: Algebra 1, which was then in development. The application was to be piloted in four California school districts during the 2010-2011 school year. HMH contracted with Empirical Education Inc. to conduct a one-year randomized experiment aimed at producing evidence of the effectiveness of HMH Fuse for increasing algebra achievement and student attitudes toward math for seventh and eighth grade students.HMH Fuse for the Apple iPad contains the content of the Holt McDougal Algebra 1 2011© text and includes interactive lessons, explanations, quizzes, and problem solving. In addition, HMH Fuse comes with the 300 videos that are also available online to students using the traditional print version of the text. We compared classes using HMH Fuse on the iPad with classes using the conventional text containing the same content. Findings: We found no impact of HMH Fuse on the primary measure of algebra achievement, the California Standards Test (CST), on average across the four districts. One of the school districts, Riverside Unified, initiated its own investigation of the data for the participating students and found what appeared to be a strong impact. We used the same statistical modeling approach to examine impacts for this district and for the other three. For the other three, and consistent with the overall results, there was no discernible difference between HMH Fuse and control. For Riverside, however, we found a substantial impact equivalent to a nine-point increase in percentile standing (p=.023). The following figure represents the differential effect of HMH Fuse in the other three districts compared to the effect in Riverside.It is also noteworthy that the teachers in Riverside were selected for the pilot on the basis of their experience with technology innovations and reported more time instructing with HMH Fuse than reported by most of the other teachers in the study.On average across the districts, we found no impact on a second measure, the End of Course Assessment. We have some confidence of a positive impact on student attitudes toward math as measured by the Student Attitude Questionnaire. It is notable that students with positive attitudes toward math were found to achieve higher scores on the CST.We also gathered implementation data via student and teacher surveys. Conditions for implementation were generally good across both groups; teachers received the necessary materials within the first few weeks of school although many teachers reported technical difficulties. We have some confidence in an impact of HMH Fuse on time spent on the algebra program outside the class, number of videos watched, and student attitude towards math. At the end of the school year, nine of the eleven teachers would choose to continue teaching with HMH Fuse over the control curriculum. Research Methods: This was a randomized control trial (RCT) in which we randomly assigned one algebra period for each participating teacher to the program condition, in which they use HMH Fuse. Each teacher’s remaining algebra sections formed the control group assigned to use the regular text version of the program. Across the four districts we had six schools and 11 teachers. In the control group there were 23 sections of Algebra 1 and 625 students with CST posttests. In HMH Fuse group there were 11 sections (one per teacher) and 318 students with CST posttests. Riverside had two teachers with seven control sections with 197 students and two HMH Fuse sections with 64 students. Because randomization was blocked by teacher, the two teachers and nine sections in Riverside constituted a very small, yet independent RCT. Statistical modeling took full advantage of the pretest and demographic information to provide appropriate controls and adjustments were made for clustering of students in sections. Conclusion: After a one-year pilot implementation with HMH Fuse, we do not have evidence of a generalizable effect of the program on algebra achievement. We did find clear evidence that the effect was dependent on local conditions. For two teachers in one school — selected for the study on the basis of experience with technology innovations — there was an impact. While we cannot generalize the results beyond these two teachers, the study is suggestive of approaches that may lead to success with applications such as HMH Fuse.