Evaluation of Educational Interventions Based on Average Treatment Effect: A Case Study

Abstract

Relative to randomized controlled trials (RCTs) with privacy and ethical concerns, observational studies are becoming dominant in education research. In an observational study, it is necessary and important to correctly evaluate the effects of different interventions (i.e., covariates) on student performance with observational data. However, these effects’ evaluation results are probably derived from biased estimations because the distributions of “control” and “treatment” student groups can hardly be equivalent to those in RCTs. Moreover, the collected covariates on possible educational interventions (i.e., treatments) may be confounded with student characteristics that are not included in the data. In this work, an estimation method based on the Rubin causal model (RCM) is proposed to calculate the average treatment effect (ATE) of different educational interventions. Specifically, with the selected covariates, the propensity score (i.e., the probability of treatment exposure conditional on covariates) is considered as a criterion to stratify the observational data into sub-classes with balanced covariate distributions between the control and treatment groups. Combined with Neyman’s estimation, the ATE of each sample is then obtained. We verify the effectiveness of this method with real observational data on student performance and its covariates.