Detecting Instruction Effects\u2014Deciding Between Covariance Analytical and Change-Score Approach

Carmen Köhler,Johannes Hartig,Alexander Naumann

doi:10.1007/s10648-020-09590-6

Carmen Köhler, Johannes Hartig + Show 1 more

Open Access

https://doi.org/10.1007/s10648-020-09590-6

Copy DOI

Abstract

The article focuses on estimating effects in nonrandomized studies with two outcome measurement occasions and one predictor variable. Given such a design, the analysis approach can be to include the measurement at the previous time point as a predictor in the regression model (ANCOVA), or to predict the change-score of the outcome variable (CHANGE). Researchers demonstrated that both approaches can result in different conclusions regarding the reported effect. Current recommendations on when to apply which approach are, in part, contradictory. In addition, they lack direct reference to the educational and instructional research contexts, since they do not consider latent variable models in which variables are measured without measurement error. This contribution assists researchers in making decisions regarding their analysis model. Using an underlying hypothetical data-generating model, we identify for which kind of data-generating scenario (i.e., under which assumptions) the defined true effect equals the estimated regression coefficients of the ANCOVA and the CHANGE approach. We give empirical examples from instructional research and discuss which approach is more appropriate, respectively.

Highlights

The article focuses on estimating effects in nonrandomized studies with two outcome measurement occasions and one predictor variable
The analysis approach can be to include the measurement at the previous time point as a predictor in the regression model (ANCOVA), or to predict the change-score of the outcome variable (CHANGE)
We refer to this approach as the CHANGE approach, which can be expressed as outcomeT 2i − outcomeT 1i 1⁄4 β0 þ β1X i þ ei: Note that throughout the article, we use the terms outcome at T1 and outcome at T2 instead of pretest and posttest to stress that the measurements of the variable of interest needs to be on the same scale at T1 and T2 in order to even apply the CHANGE approach

Summary

Introduction

The article focuses on estimating effects in nonrandomized studies with two outcome measurement occasions and one predictor variable Given such a design, the analysis approach can be to include the measurement at the previous time point as a predictor in the regression model (ANCOVA), or to predict the change-score of the outcome variable (CHANGE). The analysis approach can be to include the measurement at the previous time point as a predictor in the regression model (ANCOVA), or to predict the change-score of the outcome variable (CHANGE) Researchers demonstrated that both approaches can result in different conclusions regarding the reported effect. We use empirical examples from instructional research to illustrate the necessary considerations for making an informed decision These considerations concern the study design, the time points of the assessed variables, and possible time point–specific and time point invariant cofounders. We return to our data example to draw a conclusion, and finish with a general discussion

Objectives

Results

Conclusion