A Multiple-Design, Experimental Strategy: Academic Probation Warning Letter's Impact on Student Achievement

Abstract

We aimed to compare the findings of three research designs to bracket effect estimates of a strongly worded warning letter delivered by certified mail to students on academic probation. We embedded an experiment within a regression discontinuity design and calculated two achievement estimates, average GPA and percentage of students remaining on probation. Study participants attended a large Midwestern college. Cohen's d experimental effect size was .45. Regression discontinuity design results were validated by our experimental evidence, and outcome measures were generally statistically significant. We provided additional supportive evidence using comparative RD control group design logic. Regression point displacement design results were successfully replicated using a within-study comparison inside the experiment. In the context of probation, a diverse design, replicative approach provided considerable promise for more precise estimation of intervention effectiveness. We found no deleterious impact on reenrollment and concluded that the certified letter represents an inexpensive probation policy.