Bias and Mean Square Errors of Estimators as Criteria for Evaluating Competing Analysis Strategies in Quasi-Experiments

Abstract

The evaluation of competing analysis strategies based on estimator bias and the mean square errors of estimators is demonstrated using gains in standard scores and analysis of covariance adjusted for errors of measurement procedures for quasi-experiments conforming to the fan spread hypothesis. Some confusion in the appropriateness of these analysis procedures is resolved by considering the fan spread model defined in latent and manifest variables, large and small sample properties of the estimators, and explicitly stating the nature of individual academic growth patterns. For a linear model of individual academic growth both procedures provide an unbiased estimator with equal mean square errors when the samples are large. With small samples, analysis of covariance adjusted for errors of measurement provides an unbiased estimator, while the gain in standard scores estimator is biased and has a spuriously low mean square error. Under a nonlinear model and large samples only gains in standard scores provide an unbiased estimator. Neither procedure is appropriate for a nonlinear model with small samples. A data example is provided to demonstrate the study's findings. It is recommended that both criteria of bias and mean square errors of estimators be considered when evaluating recently developed analytic strategies for quasi-experiments.