A Methodological Review of the Pennsylvania Foreign Language Research Project

Abstract

ABSTRACT: An attempt is made to give a methodological review of the Pennsylvania Foreign Language Research Project. The history of evaluation and assessment is reviewed. The distinction between conclusion‐and decision‐oriented research is made and applied to the notion of evaluation. The implications of these notions for the design of evaluation studies are specified. These implications are employed in reviewing the research project. The main criticisms of the study concern (1) lack of random assignment of units for the main comparison, (2) possible lack of validity of the analyses of covariance, and (3) use of inappropriate analyses for testing certain hypotheses. Some commendable features of the project are (1) use of more powerful statistical techniques than are ordinarily used, (2) greater effort than that usually given to improve external and internal validity, and (3) monitoring of treatments to improve inferential validity. Given that the main press of the evidence is for no detectable differences in the treatments, it is concluded that the sensitivity of the comparisons should be ascertained in order to see if this finding is due to lack of statistical power or small true differences.