Confounding in Observational Studies using Standardized Test Data: Careful Disentanglement of Statistical Interpretations and Explanations

Abstract

Standardized testing of public school students has been, and continues to be, a focal point of the political side of education reform. The use of standardized test results to set policies and identify problem schools requires some understanding of what factors contribute to high or low overall scores. Many large datasets at the state and national levels contain information about mean scores (i.e., average class size, poverty level, teacher salary, etc.); it is tempting to create statistical models, draw cause-and-effect conclusions, and perhaps set policy based on statistically significant relationships observed in these data; for example, National Assessment of Educational Progress (NAEP, US National Center for Education Statistics, n.d.); Trends in International Mathematics and Science Study, and Progress in International Reading Literacy Study (TIMSS & PIRLS International Study Center, n.d.); Programme for International Student Assessment (PISA, Organisation for Economic Co-operation and Development, n.d.). However, many examples of confounding, that is, the apparent associations between the variables that change depending on which covariates are selected, can be found in these associations. In this chapter, some results using standardized testing data are presented, which demonstrate by example the difficulties inherent in making conclusions or comparisons based on observational data and disentangling second- and third-order influences on these relationships. First, some statistical terminology is reviewed, and some simplified, fabricated examples are presented to illustrate concepts. Next, a dataset containing scores for the Illinois Standardized Achievement Test (ISAT, Illinois State Board of Education, n.d.), taken by Grade 8 students in Illinois public schools, is used to demonstrate confounding relationships. Finally, the Scholastic Achievement Test (SAT, College Board, n.d.) scores by state are used to show some misleading rankings of states’ average scores.